Ab Test Setup

What it does

When your agent uses it

• Setting up an A/B test
• Calculating sample size
• Designing an experiment
• Analyzing results

Example prompts

• “/ab-test-setup”

Requirements

• Python 3

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Define hypothesis and success metric
2. 2Run sample_size_calculator.py with baseline conversion and minimum detectable effect
3. 3Create test configuration JSON (see Common Patterns)
4. 4Run test_designer.py to generate complete test plan
5. 5Share plan with stakeholders for alignment before launch

What it can do on your machine

Read from SKILL.md and the folder at commit 4a698e8. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 3 files in scripts/ (Python), which the agent can run.

  Shell commands in SKILL.md call:

  • python

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Ab Test Setup loads about 1.2k tokens when it runs, and up to ~2.6k if it reads all its reference files. Until then it costs about 58 tokens; SKILL.md has 395 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~58

When it runs · the whole SKILL.md, loaded when a task matches

~1.2k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~2.6k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from borghei/Claude-Skills at commit 4a698e8, republished under its MIT licence (© borghei). 395 words, ~1,198 tokens.

name

ab-test-setup

description

Design and analyze A/B tests: sample size, test duration, and statistical significance for conversion experiments. Use when setting up an A/B test, calculating sample size, designing an experiment, or analyzing results.

license

MIT + Commons Clause

metadata.version

1.0.0

metadata.author

borghei

metadata.category

marketing

metadata.domain

experimentation

metadata.updated

2026-04-02

metadata.tags

ab-testing, experimentation, statistics, sample-size, conversion-rate

# Calculate required sample sizes for a test
python scripts/sample_size_calculator.py --baseline 0.05 --mde 0.10 --power 0.80

# Design a complete A/B test plan
python scripts/test_designer.py test_config.json

# Analyze A/B test results
python scripts/results_analyzer.py results.json

{
  "test_name": "Homepage CTA Button Color",
  "hypothesis": "Changing the CTA button from blue to green will increase click-through rate",
  "metric_primary": "cta_click_rate",
  "metric_secondary": ["signup_rate", "bounce_rate"],
  "baseline_rate": 0.045,
  "minimum_detectable_effect": 0.10,
  "significance_level": 0.05,
  "power": 0.80,
  "variants": [
    {"name": "control", "description": "Current blue CTA button"},
    {"name": "treatment", "description": "Green CTA button"}
  ],
  "daily_traffic": 5000,
  "allocation": {"control": 0.50, "treatment": 0.50}
}

{
  "test_name": "Homepage CTA Button Color",
  "variants": {
    "control": {"visitors": 12500, "conversions": 563},
    "treatment": {"visitors": 12500, "conversions": 625}
  },
  "metric": "cta_click_rate",
  "significance_level": 0.05
}

© borghei, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 5 other files (scripts, references) in marketing/ab-test-setup of borghei/Claude-Skills.

• SKILL.md
• examples/test_results.csv
• references/ab-testing-guide.md
• scripts/results_analyzer.py
• scripts/sample_size_calculator.py
• scripts/test_designer.py

Open the folder on GitHubat commit 4a698e8