Agent skill

Product Experimentation

by magnus919 in magnus919/agent-skills

Run end-to-end product experiments from assumption to decision: translate assumptions into testable hypotheses and experiment briefs, select the right method among qualitative interviews…

MITAuto-check passedMarketing & SEO

Install Product Experimentation

skills CLI
$ npx skills add magnus919/agent-skills --skill product-experimentation -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install magnus919/agent-skills product-experimentation --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/magnus919/agent-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/product-experimentation .claude/skills/product-experimentation && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
product-experimentation
GitHub stars
115
Token cost
~2.6k tokens
SKILL.md length
925 words
Files
11 (incl. references)
Skills in repo
131
Repo updated
First seen
Licence
MIT

At a glance

Run end-to-end product experiments from assumption to decision: translate assumptions into testable hypotheses and experiment briefs, select the right method among qualitative interviews…

  • Works in 8 steps: Map assumptions → Translate into hypotheses → Select the appropriate method → …
  • Prototype test is the clearly right answer without statistical measurement
  • SKILL.md covers Pipeline, Loading Guide, Working Method and Trigger Conditions, plus 2 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Product Experimentation is an agent skill from magnus919/agent-skills. Run end-to-end product experiments from assumption to decision: translate assumptions into testable hypotheses and experiment briefs, select the right method among qualitative interviews, prototypes, concierge tests, fake doors, feature flags, and A/B tests, and produce readouts that update the roadmap and decision record. Do not use when a qualitative or prototype test is the clearly right answer without statistical measurement; do not prescribe A/B testing by default; do not treat statistical significance as…

Its SKILL.md is about 2.6k tokens, which your agent loads only when the skill is triggered. The skill folder holds 13 other files, including reference files (for example `README.md`, `evals/evals.json` and `references/discovery-brief.md`). Compatibility notes: Agent-agnostic — works with any agent framework supporting the Agent Skills format. No external services, proprietary tools, or runtime dependencies required.

It sits in Marketing & SEO, covering A/B testing. The repository describes itself as: Curated collection of AI agent skills for Hermes and other agent frameworks. The licence is MIT.

When your agent uses it

  • Prototype test is the clearly right answer without statistical measurement
  • Do not prescribe A/B testing by default
  • Do not treat statistical significance as the only decision criterion
  • Hide ethical and guardrail considerations

Example prompts

  • “/product-experimentation”

Requirements

  • Compatibility (from SKILL.md): Agent-agnostic — works with any agent framework supporting the Agent Skills format. No external services, proprietary tools, or runtime dependencies required.

Workflow steps

8 steps, taken from the step headings in SKILL.md.

  1. Map assumptions
  2. Translate into hypotheses
  3. Select the appropriate method
  4. Define instrumentation, guardrails, and ethics
  5. Determine exposure and duration
  6. Run and monitor
  7. Decide
  8. Record the readout

What it can do on your machine

Read from SKILL.md and the folder at commit 22b4723. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

  • Compatibility

    Agent-agnostic — works with any agent framework supporting the Agent Skills format. No external services, proprietary tools, or runtime dependencies required.

    From compatibility in the SKILL.md frontmatter.

Context cost

Product Experimentation loads about 2.6k tokens when it runs, and up to ~7.3k if it reads all its reference files. Until then it costs about 153 tokens; SKILL.md has 925 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~153
When it runs · the whole SKILL.md, loaded when a task matches
~2.6k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~7.3k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from magnus919/agent-skills at commit 22b4723, republished under its MIT licence (© magnus919). 925 words, ~2,571 tokens.

Download SKILL.mdSave it as .claude/skills/product-experimentation/SKILL.md (or your agent's skills folder). This skill also uses 10 other files; get the full folder from GitHub.
name
product-experimentation
description
Run end-to-end product experiments from assumption to decision: translate assumptions into testable hypotheses and experiment briefs, select the right method among qualitative interviews, prototypes, concierge tests, fake doors, feature flags, and A/B tests, and produce readouts that update the roadmap and decision record. Do not use when a qualitative or prototype test is the clearly right answer without statistical measurement; do not prescribe A/B testing by default; do not treat statistical significance as the only decision criterion or hide ethical and guardrail considerations.
compatibility
Agent-agnostic — works with any agent framework supporting the Agent Skills format. No external services, proprietary tools, or runtime dependencies required.
license
MIT
metadata.tags
product-experimentation, hypothesis-testing, ab-testing, feature-flags, experiment-design, product-decisions, guardrails, statistical-validity

Product Experimentation

End-to-end product experimentation: from assumption mapping through method selection, instrumentation, guardrail enforcement, and decision-readout that updates the product roadmap. Owns the complete experiment workflow; routes statistical design and rollout mechanics to specialist skills.

Pipeline

ASSUMPTIONS → [HYPOTHESIS] → [METHOD SELECT] → [INSTRUMENT] → [RUN] → [DECIDE] → [RECORD]
                  |                |                |           |          |           |
             Experiment       Qualitative       Tracking     Guardrail   Decision    Readout
               brief          Prototype          plan         monitor     rules      learning
                              Operational
                              Quantitative

Loading Guide

Load only the reference or template relevant to the task. Do not load every file at once.

FileLoad when
references/discovery-brief.mdYou need to understand how experimentation concepts map across skills and where this skill's boundaries are
references/method-selection.mdChoosing among qualitative, prototype, operational, and quantitative test methods
references/guardrails-and-ethics.mdDefining guardrail metrics, ethical boundaries, stopping rules, and decision ownership
references/experiment-readout.mdProducing a decision-impact readout that updates the roadmap or decision record
templates/experiment-brief.mdFilling out a structured experiment brief from an assumption
templates/assumption-map.mdMapping assumptions to risk, evidence, and testability before designing experiments
templates/guardrail-and-decision-rule.mdRecording guardrails, stopping rules, and decision criteria for an experiment
templates/readout-learning-entry.mdDocumenting experiment outcome and updating the roadmap, decision log, or lifecycle evidence

Working Method

1. Map assumptions

Surface the assumptions driving the proposed change. Classify each by risk (what breaks if it is wrong), evidence strength (what evidence already exists), and testability (can it be tested, and how cheaply). Use templates/assumption-map.md.

2. Translate into hypotheses

Convert the riskiest, least-evidenced assumptions into falsifiable hypotheses. Each hypothesis names the independent variable (what changes), the dependent variable (what outcome is measured), the predicted direction, and the smallest effect that matters. Use templates/experiment-brief.md.

3. Select the appropriate method

Choose the lightest-weight method that can falsify the hypothesis with sufficient confidence. The method ladder, from lightest to heaviest:

MethodBest forCostStatistical rigor
Qualitative interviewsUncovering unknown unknowns, mental models, problem validationLowestNone (descriptive)
Prototype testsInteraction flow, usability, concept validationLowNone (observational)
Concierge testsValue delivery, willingness to pay, operational feasibilityLow-MediumNone (manual)
Fake doorsDemand signals, willingness to click/commitMediumLow (conversion rate only)
Feature flagsOperational safety, incremental rollout, kill-switchMediumMedium (controlled rollout)
A/B testsCausal attribution of a specific change to a metricHighHigh (randomized controlled)

Do not default to A/B testing. Start at the top of the ladder and only move down when the question cannot be answered at the current level. A qualitative interview or prototype test is often the right answer. Full method selection guidance is in references/method-selection.md.

4. Define instrumentation, guardrails, and ethics

Before running the experiment, define:

  • Instrumentation: what metrics are tracked, how they are computed, and that they are measurable with the available tooling. Route measurement contracts to product-analytics-and-measurement.
  • Guardrails: mandatory safety metrics that can stop the experiment regardless of the primary outcome. At minimum: error rate, latency/degradation, and any domain-specific harm metric. Every experiment must name at least one guardrail metric. See references/guardrails-and-ethics.md.
  • Ethical boundaries: user consent, data minimization, vulnerable-population considerations, and institutional-review alignment. Record all ethical decisions.
  • Stopping rules: when the experiment stops early — guardrail breach, sufficient evidence reached, or time cap reached.
  • Decision ownership: who makes the ship/no-ship call and what inputs they consider (statistical evidence, guardrail evidence, qualitative signal, practical constraints).

Use templates/guardrail-and-decision-rule.md to record these.

5. Determine exposure and duration

Define the target population, allocation, and minimum detectable effect. Route statistical design (power analysis, sample-size calculation, estimator selection) to ../data-scientist/SKILL.md. An underpowered experiment — one that cannot detect the smallest effect that matters — is a validity failure; do not ship based on a null result from an underpowered test.

Show full SKILL.md (370 more words)Show less
6. Run and monitor

Execute the experiment. Monitor guardrails continuously. Route production rollout mechanics (feature flags, canary stages, progressive delivery) to ../release-engineering/SKILL.md.

7. Decide

Make the ship/no-ship decision using multiple criteria, never statistical significance alone:

CriterionWeightSource
Statistical evidenceRequireddata-scientist
Practical significanceRequiredIs the effect large enough to matter?
Guardrail evidenceBlockingAll guardrails must pass
Qualitative evidenceInformativeUser feedback, support tickets
ReversibilityInformativeCan we undo this if wrong?
Opportunity costInformativeWhat else could we build instead?

A statistically significant result with a failing guardrail is a no-ship. A statistically significant result that exceeds authority boundaries (e.g., safety, compliance, ethics) is a no-ship. Record the decision and its rationale.

8. Record the readout

Document what was learned and what changed as a result. The readout updates the product roadmap, backlog, decision log, or lifecycle evidence. Routing: feeds product-roadmapping-and-portfolio (roadmap updates), product-adoption (adoption evidence), and product-lifecycle-learning (retained learning). Use templates/readout-learning-entry.md and references/experiment-readout.md.

Trigger Conditions

Load this skill when:

  • The task involves designing, running, or deciding on a product experiment
  • You need to choose between qualitative, prototype, operational, and quantitative test methods
  • You have assumptions that need to be tested before committing to build
  • You need to define guardrails, stopping rules, or decision criteria for an experiment
  • You need to interpret experiment results and make a ship/no-ship decision
  • You need to record experiment outcomes that update product direction

When Not to Use

  • Statistical design (power analysis, estimator selection, significance testing in depth) — route to data-scientist. This skill frames the question and selects the method; data-scientist owns the statistical machinery.
  • Production rollout mechanics (feature-flag infrastructure, canary stages, CD pipeline integration) — route to release-engineering. This skill defines the experiment design; release-engineering owns the safe delivery.
  • User research and usability testing — route to product-design-and-ux for interaction-focused studies.
  • Pricing-specific tests — route to financial-modeling for elasticity, willingness-to-pay, and pricing-page experiments.
  • Opportunity-solution tree construction — route to product-methodology for connecting customer needs to build decisions before experimentation.
  • Pure analytics instrumentation (tracking-plan design, event taxonomy, metric definitions) — route to product-analytics-and-measurement for measurement contracts.

Portability

This skill is intentionally host-neutral. It requires no profile system, output format, scripts, or external services. Load references and templates directly by path using the host agent's normal file-loading mechanism.

© magnus919, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 10 other files (references) in product-experimentation of magnus919/agent-skills.

  • SKILL.md
  • README.md
  • evals/evals.json
  • references/discovery-brief.md
  • references/experiment-readout.md
  • references/guardrails-and-ethics.md
  • references/method-selection.md
  • templates/assumption-map.md
  • templates/experiment-brief.md
  • templates/guardrail-and-decision-rule.md
  • templates/readout-learning-entry.md

Open the folder on GitHubat commit 22b4723

Compare with similar skills

Product Experimentation next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Product Experimentation compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Product Experimentation this skillmagnus919/agent-skills115—~2.6kAutomated safety check: PassMIT
Experimentationandreaskelm/pm-brain234—~2.3kAutomated safety check: PassCustom licence
Experiment Designrampstackco/claude-skills945—~7.9kAutomated safety check: PassMIT
Ab Test Results Readouthashgraph-online/awesome-codex-plugins1.3k—~859Automated safety check: PassMIT
Ab Testingcoreyhaines31/marketingskills54k3 repos~3.1kAutomated safety check: PassMIT
AnalyticsNexus-JPF/note-companion8707 repos~2.2kAutomated safety check: PassMIT

Similar skills

  • Experimentation

    andreaskelm/pm-brain

    Design and run product experiments at a practical PM level — A/B tests, hypothesis tests, rollouts, feature flags, and reading results without pretending to be a statistician.

    234 GitHub stars~2.3k tokensUpdated 2 days ago
    Marketing & SEOAuto-check passed
  • Experiment Design

    rampstackco/claude-skills

    A discipline for designing experiments (A/B tests, multivariate, holdouts) so the results actually answer the question you asked.

    945 GitHub stars~7.9k tokensUpdated 4 days ago
    Marketing & SEOAuto-check passed
  • Ab Test Results Readout

    hashgraph-online/awesome-codex-plugins

    Analyze and communicate A/B test results with metric readouts, subgroup analysis, data-quality checks, ad hoc investigation, visualization, and launch recommendations.

    1.3k GitHub stars~859 tokensUpdated yesterday
    Marketing & SEOAuto-check passed
  • Ab Testing

    coreyhaines31/marketingskills

    When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.

    54k GitHub starsUsed in 3 repos~3.1k tokens
    Marketing & SEOAuto-check passed
  • Analytics

    Nexus-JPF/note-companion

    When the user wants to set up, improve, or audit analytics tracking and measurement.

    870 GitHub starsUsed in 7 repos~2.2k tokens
    Marketing & SEOAuto-check passed
  • Ad Test Designer

    aaron-he-zhu/aaron-marketing-skills

    A skill your agent uses when the user asks to "design an A/B test", "set up a creative/landing test", "run an incrementality test", or "is this result statistically and practically material?"…

    2.9k GitHub starsUsed in 2 repos~2.8k tokens
    Marketing & SEOAuto-check passed

More from magnus919/agent-skills

All 131 skills in this repo
  • Artifact Pyramids

    magnus919/agent-skills

    Organize durable agent research outputs as summaries, analysis, and evidence dossiers.

    119 GitHub stars~2.7k tokensUpdated today
    Auto-check passed
  • Ascii City Engine

    magnus919/agent-skills

    Build portable, first-person colored ASCII city engines and small GIS-derived city packs.

    119 GitHub stars~1.1k tokensUpdated today
    Auto-check passed
  • Color Management

    magnus919/agent-skills

    Manage color workflows with ICC profiles, working spaces, gamut mapping, and color science.

    119 GitHub stars~2.6k tokensUpdated today
    Auto-check: notes
  • Data Scientist

    magnus919/agent-skills

    A skill your agent uses for PhD-level expertise in data science, statistics, and machine learning: rigorous statistical analysis, experimental design, causal inference, advanced modeling, research…

    119 GitHub stars~4.1k tokensUpdated today
    Auto-check passed
  • Docker Compose

    magnus919/agent-skills

    Use Docker Compose to define, run, debug, and harden multi-container applications.

    119 GitHub stars~2k tokensUpdated today
    Auto-check: notes
  • Fpga Development

    magnus919/agent-skills

    Design, review, simulate, and verify FPGA logic using explicit RTL contracts, clock and reset models, CDC analysis, timing constraints, and reproducible implementation evidence.

    119 GitHub stars~2.7k tokensUpdated today
    Auto-check passed

Questions about Product Experimentation

What does Product Experimentation do?

Run end-to-end product experiments from assumption to decision: translate assumptions into testable hypotheses and experiment briefs, select the right method among qualitative interviews…. Product Experimentation is an agent skill from magnus919/agent-skills. Run end-to-end product experiments from assumption to decision: translate assumptions into testable hypotheses and experiment briefs, select the right method among qualitative interviews, prototypes, concierge tests, fake doors, feature flags, and A/B tests, and produce readouts that update the roadmap and decision record.

When should I use Product Experimentation?

Product Experimentation fits situations like: prototype test is the clearly right answer without statistical measurement; do not prescribe A/B testing by default; do not treat statistical significance as the only decision criterion; hide ethical and guardrail considerations.

How do I install Product Experimentation in Claude Code?

Run `npx skills add magnus919/agent-skills --skill product-experimentation -a claude-code`. Or copy the skill folder (product-experimentation in magnus919/agent-skills) into .claude/skills/product-experimentation in your project. Claude Code loads it when a task matches its description.

How do I install Product Experimentation in Codex?

Run `npx skills add magnus919/agent-skills --skill product-experimentation -a codex`. Or copy the skill folder (product-experimentation in magnus919/agent-skills) into .agents/skills/product-experimentation in your project. Codex loads it when a task matches its description.

Can I use Product Experimentation in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add magnus919/agent-skills --skill product-experimentation -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/product-experimentation, .gemini/skills/product-experimentation, .github/skills/product-experimentation and .opencode/skills/product-experimentation in your project.

What does Product Experimentation need to run?

SKILL.md names no scripts, command-line tools or credentials: Product Experimentation is instructions for the agent only. Compatibility (from SKILL.md): Agent-agnostic — works with any agent framework supporting the Agent Skills format. No external services, proprietary tools, or runtime dependencies required..

Does Product Experimentation access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Product Experimentation safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Product Experimentation use?

Product Experimentation is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Product Experimentation use?

About 2.6k tokens (SKILL.md is roughly 10k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 4.7k tokens, read only when the agent opens those files.

What are the alternatives to Product Experimentation?

Skills that share tags, products or a category with Product Experimentation: Experimentation (andreaskelm/pm-brain, 234 stars), Experiment Design (rampstackco/claude-skills, 945 stars), Ab Test Results Readout (hashgraph-online/awesome-codex-plugins, 1.3k stars) and Ab Testing (coreyhaines31/marketingskills, 54k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Product Experimentation?

magnus919 (a GitHub user) maintains it in magnus919/agent-skills, which has 115 GitHub stars. The repository holds 131 skills in this directory. The repository was last updated on October 10, 2026.

Source: magnus919/agent-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.