Stat Ab Testing

What it does

When your agent uses it

• The user needs to set up an experiment
• Calculate required sample size
• Interpret test results
• Decide between testing methodologies — even if they say should we A/B test this

Example prompts

• “should we A/B test this”
• “how many users do we need”
• “is the test result conclusive”
• “/stat-ab-testing”

Workflow steps

7 steps, taken from the first numbered list in SKILL.md.

1. 1Hypothesis: What do you expect to happen and why?
2. 2Primary metric: ONE key metric (conversion, revenue, retention)
3. 3Guardrail metrics: Metrics that must NOT degrade (page load time, error rate)
4. 4Randomization unit: User, session, or device?
5. 5Sample size: Calculated from baseline, MDE, α, power
6. 6Duration: Account for weekly cycles (minimum 1-2 full weeks)
7. 7Stopping rules: Pre-defined — do NOT peek and stop early without correction

What it can do on your machine

Read from SKILL.md and the folder at commit 4e7f4f8. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  No scripts in the folder and no shell commands in SKILL.md (its code samples are markdown).

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Stat Ab Testing loads about 1k tokens when it runs, and up to ~5.9k if it reads all its reference files. Until then it costs about 122 tokens; SKILL.md has 332 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~122

When it runs · the whole SKILL.md, loaded when a task matches

~1k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~5.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from asgard-ai-platform/skills at commit 4e7f4f8, republished under its MIT licence (© asgard-ai-platform). 332 words, ~1,049 tokens.

name

stat-ab-testing

description

Design and analyze A/B tests with proper statistical methodology including sample size calculation, randomization, frequentist and Bayesian approaches, and sequential testing. Use this skill when the user needs to set up an experiment, calculate required sample size, interpret test results, or decide between testing methodologies — even if they say 'should we A/B test this', 'how many users do we need', 'is the test result conclusive', or 'can we stop the test early'.

metadata.category

WP-21 設計/資訊/傳播/公衛

metadata.tags

statistics, ab-testing, experimentation

IRON LAW: Calculate Sample Size BEFORE Running the Test

Running a test without knowing the required sample size leads to two
failures: stopping too early (false positives) or running too long (waste).

Required inputs: baseline conversion rate, minimum detectable effect (MDE),
significance level (α), power (1-β). Calculate BEFORE starting.

n per group ≈ (Z_α/2 + Z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²

# A/B Test Design: {Experiment Name}

## Hypothesis
- H₀: {no difference}
- H₁: {expected improvement}
- Primary metric: {metric}
- MDE: {X% relative}

## Sample Size
- Baseline rate: {X%}
- Required N per group: {N}
- Estimated duration: {days/weeks}

## Results (post-test)
| Metric | Control | Treatment | Diff | CI (95%) | p-value |
|--------|---------|-----------|------|----------|---------|
| {primary} | X% | X% | +X% | [X, X] | {value} |

## Decision
{Ship / Don't ship / Extend test} — {rationale}

© asgard-ai-platform, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 3 other files (references) in stat-ab-testing of asgard-ai-platform/skills.

• SKILL.md
• examples/sample_scenario.md
• references/bandits.md
• references/bayesian-ab.md

Open the folder on GitHubat commit 4e7f4f8