New: 97,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Ab Test Setup

by sickn33 in sickn33/agentic-awesome-skills

A skill your agent uses when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.

MITAuto-check passedMarketing & SEO

Install Ab Test Setup

skills CLI
$ npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install sickn33/agentic-awesome-skills ab-test-setup --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/sickn33/agentic-awesome-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/ab-test-setup .claude/skills/ab-test-setup && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
ab-test-setup
GitHub stars
47k
Used in
1 other repo
Token cost
~2.2k tokens
SKILL.md length
977 words
Files
1
Skills in repo
1,394
Repo updated
First seen
Licence
MIT

At a glance

A skill your agent uses when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.

  • Works in 5 steps: Event firing: Trigger each event the… → Variant attribution: Verify that the… → De-duplication: Confirm that a user… → …
  • Designing an A/B
  • SKILL.md covers 1️⃣ Purpose & Scope, 2️⃣ Pre-Requisites, 3️⃣ Hypothesis Lock (Hard Gate) and 4️⃣ Assumptions & Validity…, plus 12 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Ab Test Setup is an agent skill from sickn33/agentic-awesome-skills. Use when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.

Its SKILL.md is about 2.2k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in Marketing & SEO, covering A/B testing and Experimental design. The repository describes itself as: AAS Core is the local, agent-first control plane for complete catalog discovery, agent-owned selection, stack validation, and planning, backed by 2,400+ agentic skills. Includes… The licence is MIT.

When your agent uses it

  • Designing an A/B
  • Split test: define the hypothesis
  • Control and variants
  • Estimate sample size

Example prompts

  • “/ab-test-setup”

Requirements

  • Python 3

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

  1. Event firing: Trigger each event the primary and secondary metrics depend on (sign-up, add-to-cart, custom event) on staging or a debug…
  2. Variant attribution: Verify that the variant assignment ID is attached to every fired event — not just the entry event. Use your…
  3. De-duplication: Confirm that a user reloading the page does not cause double-counted events. Use a stable event/transaction ID and…
  4. Sample randomization: Check sample-ratio mismatch against the configured allocation with a pre-specified statistical check and adequate…
  5. Guardrail metric pipeline: Each guardrail metric defined in §6️⃣ must have a working dashboard or alert by the time the test launches.

What it can do on your machine

Read from SKILL.md and the folder at commit 1e53ce2. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • statsmodels.org

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Ab Test Setup loads about 2.2k tokens when it runs. Until then it costs about 45 tokens; SKILL.md has 977 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~45
When it runs · the whole SKILL.md, loaded when a task matches
~2.2k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from sickn33/agentic-awesome-skills at commit 1e53ce2, republished under its MIT licence (© sickn33). 977 words, ~2,232 tokens.

Download SKILL.mdSave it as .claude/skills/ab-test-setup/SKILL.md (or your agent's skills folder).
name
ab-test-setup
description
Use when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.
risk
critical
source
community
date_added
2026-02-27

A/B Test Setup

1️⃣ Purpose & Scope

Define an experiment that can answer a specific product question, and verify its assumptions before exposing users. This procedure cannot guarantee validity by itself.

  • Documents the stopping rule
  • Estimates sample needs under stated assumptions
  • Makes the hypothesis and decision criteria reviewable

2️⃣ Pre-Requisites

You must have:

  • A clear user problem
  • Access to an analytics source
  • Roughly estimated traffic volume
Hypothesis Quality Checklist

A valid hypothesis includes:

  • Observation or evidence
  • Single, specific change
  • Directional expectation
  • Defined audience
  • Measurable success criteria

3️⃣ Hypothesis Lock (Hard Gate)

Before designing variants or metrics, you MUST:

  • Present the final hypothesis
  • Specify:
    • Target audience
    • Primary metric
    • Expected direction of effect
    • Minimum Detectable Effect (MDE)

Use the hypothesis already agreed in the task. If a launch-critical choice is missing, present the concrete choice for confirmation while continuing independent analysis. Do not repeatedly request approval for a decision already authorized.

4️⃣ Assumptions & Validity Check (Mandatory)

Explicitly list assumptions about:

  • Traffic stability
  • User independence
  • Metric reliability
  • Randomization quality
  • External factors (seasonality, campaigns, releases)

If assumptions are weak or violated:

  • Warn the user
  • Recommend delaying or redesigning the test

5️⃣ Test Type Selection

Choose the simplest valid test:

  • A/B Test – single change, two variants
  • A/B/n Test – multiple variants, higher traffic required
  • Multivariate Test (MVT) – interaction effects, very high traffic
  • Split URL Test – major structural changes

Default to A/B unless there is a clear reason otherwise.

6️⃣ Metrics Definition

Primary Metric (Mandatory)
  • Single metric used to evaluate success
  • Directly tied to the hypothesis
  • Pre-defined and frozen before launch
Secondary Metrics
  • Provide context
  • Explain why results occurred
  • Must not override the primary metric
Guardrail Metrics
  • Metrics that must not degrade
  • Used to prevent harmful wins
  • Trigger test stop if significantly negative

7️⃣ Sample Size & Duration

Define upfront:

  • Baseline rate
  • MDE
  • Significance level alpha (often 0.05, corresponding to 95% confidence)
  • Statistical power (typically 80%)

Estimate:

  • Required sample size per variant
  • Expected test duration

Do NOT proceed without a realistic sample size estimate.

Tracking Verification (Required before Gate 8)

Before entering the Execution Readiness Gate below, run through this checklist to make "Tracking is verified" mean something concrete:

  1. Event firing: Trigger each event the primary and secondary metrics depend on (sign-up, add-to-cart, custom event) on staging or a debug page, and confirm it arrives within that pipeline’s documented latency; record the observed delay.
  2. Variant attribution: Verify that the variant assignment ID is attached to every fired event — not just the entry event. Use your analytics' raw event view to compare a sample of 5+ events per variant.
  3. De-duplication: Confirm that a user reloading the page does not cause double-counted events. Use a stable event/transaction ID and document cross-client/server deduplication; a variant label alone is not a unique event key.
  4. Sample randomization: Check sample-ratio mismatch against the configured allocation with a pre-specified statistical check and adequate records. A fixed ±5% band on 100 records is not a valid universal randomization test. Inspect assignment stability, unit independence and missing exposure records.
  5. Guardrail metric pipeline: Each guardrail metric defined in §6️⃣ must have a working dashboard or alert by the time the test launches.

If any of the above fails, stop and resolve it before Gate 8.

8️⃣ Execution Readiness Gate (Hard Stop)

You may proceed to implementation only if all are true:

  • Hypothesis is locked
  • Primary metric is frozen
  • Sample size is calculated
  • Test duration is defined
  • Guardrails are set
  • Tracking is verified

If any item is missing, stop and resolve it.

Running the Test

Show full SKILL.md (397 more words)Show less
During the Test

DO:

  • Monitor technical health
  • Document external factors

DO NOT:

  • Stop early due to “good-looking” results
  • Change variants mid-test
  • Add new traffic sources
  • Redefine success criteria

Analyzing Results

Analysis Discipline

When interpreting results:

  • Do NOT generalize beyond the tested population
  • Do NOT claim causality beyond the tested change
  • Do NOT override guardrail failures
  • Separate statistical significance from business judgment
Interpretation Outcomes
ResultAction
Significant positiveConsider rollout
Significant negativeReject variant, document learning
InconclusiveReport uncertainty; use the pre-specified continuation rule or design a new test
Guardrail failureDo not ship, even if primary wins

Documentation & Learning

Test Record (Mandatory)

Document:

  • Hypothesis
  • Variants
  • Metrics
  • Sample size vs achieved
  • Results
  • Decision
  • Learnings
  • Follow-up ideas

Store records in a shared, searchable location to avoid repeated failures.

Refusal Conditions (Safety)

Refuse to proceed if:

  • Baseline rate is unknown and cannot be estimated
  • Traffic is insufficient to detect the MDE
  • Primary metric is undefined
  • Multiple variables are changed without proper design
  • Hypothesis cannot be clearly stated

Explain why and recommend next steps.

Key Principles (Non-Negotiable)

  • One hypothesis per test
  • One primary metric
  • Commit before launch
  • No peeking
  • Learning over winning
  • Statistical rigor first

When to Use

Use when a product change has enough eligible traffic for a randomized comparison and a measurable outcome. For low-volume launches or qualitative discovery, consider usability research or descriptive measurement instead of claiming causal lift.

Sample-size calculation example

For an illustrative binary metric, estimate the per-variant sample for a change from 10% to 11% (one percentage point, 10% relative lift), 50/50 allocation, two-sided alpha 0.05 and power 0.80. This Python 3 large-sample approximation uses Cohen's proportion effect size:

python
from math import asin, ceil, sqrt
from statistics import NormalDist

baseline, variant = 0.10, 0.11  # illustrative assumptions, not measured data
alpha, power = 0.05, 0.80
h = abs(2 * asin(sqrt(variant)) - 2 * asin(sqrt(baseline)))
z = NormalDist()
per_variant = ceil(2 * (z.inv_cdf(1 - alpha / 2) + z.inv_cdf(power)) ** 2 / h ** 2)
print(per_variant)

Expected output: 14745 observations per variant for these assumptions.

This calculation assumes independent units, one binary outcome, a fixed horizon and no multiplicity adjustment. It is inappropriate for clustered or repeated observations, sequential decisions or continuous revenue metrics. Account for eligible traffic, attrition, outcome delay and the sampling unit before turning a sample estimate into calendar duration. Equal assumed rates have zero effect size and no finite sample for detecting that difference.

Worked example

text
Observation: users abandon a long signup form.
Change: remove one optional field; unit: account; allocation: 50/50 and stable.
Primary metric: completed signup / eligible assigned accounts within 24 hours.
Guardrails: validation failures and support requests.
Before launch: estimate sample needs from baseline and MDE, verify exposure and
completion IDs, define analysis window and stopping rule.
Expected report: counts, absolute/relative effect, interval, data-quality checks,
guardrail results and a decision with its limits; never just “p < 0.05, ship”.

Limitations

  • Clustered users, spillovers and repeated observations can invalidate independent-sample calculations.
  • Sequential monitoring needs a planned sequential method; fixed-horizon significance does not authorize repeated peeking.
  • A tracking gap or sample-ratio mismatch can invalidate inference despite a favorable primary metric.
  • This skill does not activate flags, publish variants or establish regulatory compliance automatically.

© sickn33, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/ab-test-setup of sickn33/agentic-awesome-skills.

Open the folder on GitHubat commit 1e53ce2

Used in 1 other repository

We found 15 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 1 other GitHub owner. This page covers the copy in sickn33/agentic-awesome-skills, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Ab Test Setup next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Ab Test Setup compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Ab Test Setup this skillsickn33/agentic-awesome-skills47k1 repos~2.2kAutomated safety check: PassMIT
Ab Test Analyzeririnabuht12-oss/marketing-skills3.8k—~1.4kAutomated safety check: PassNone
Define Hypothesisproduct-on-purpose/pm-skills713—~966Automated safety check: PassApache-2.0
A B Test DesignOwl-Listener/designer-skills2.9k1 repos~472Automated safety check: PassMIT
Send Experiment Designeraaron-he-zhu/aaron-marketing-skills2.9k2 repos~4.1kAutomated safety check: PassApache-2.0
Content Experimentation Best Practicessanity-io/agent-toolkit1871 repos~469Automated safety check: PassMIT

Similar skills

  • Ab Test Analyzer

    irinabuht12-oss/marketing-skills

    Statistical significance calculator for A/B test results with sample size requirements, segment breakdowns, and hypothesis generation.

    3.8k GitHub stars~1.4k tokensUpdated 13 days ago
    Marketing & SEOAuto-check passed

  • Define Hypothesis

    product-on-purpose/pm-skills

    Defines a testable hypothesis with clear success metrics and a validation approach.

    713 GitHub stars~966 tokensUpdated 2 days ago
    Marketing & SEOAuto-check passed

  • A B Test Design

    Owl-Listener/designer-skills

    Design an A/B experiment — hypothesis, variants, primary metric, and sample size.

    2.9k GitHub starsUsed in 1 repo~472 tokens
    Marketing & SEOAuto-check passed

  • Send Experiment Designer

    aaron-he-zhu/aaron-marketing-skills

    A skill your agent uses when the user asks to "design an email A/B test", "set up a multivariate subject/CTA test", "run a send-time test", "build a hold-out group", or "is this email result…

    2.9k GitHub starsUsed in 2 repos~4.1k tokens
    Marketing & SEOAuto-check passed

  • Content Experimentation Best Practices

    sanity-io/agent-toolkit

    Official

    Content experimentation and A/B testing guidance covering experiment design, hypotheses, metrics, sample size, statistical foundations, CMS-managed variants, and common analysis pitfalls.

    187 GitHub starsUsed in 1 repo~469 tokens
    Marketing & SEOAuto-check passed

  • Ads Test

    AgriciDaniel/claude-ads

    Design and evaluate paid-ad experiments with hypotheses, randomization units, sample-size and duration assumptions, guardrails, platform experiment tools, analysis, and decision rules.

    9.8k GitHub stars~312 tokensUpdated 4 days ago
    Marketing & SEOAuto-check passed

More from sickn33/agentic-awesome-skills

All 1,394 skills in this repo

  • Liuguang Banlan UI

    sickn33/agentic-awesome-skills

    Implements an interface in one of two named color modes, iridescent white or colorful black, from a parameterized starter that reports measured color intensity.

    47k GitHub starsUsed in 1 repo~2.5k tokens
    Auto-check passed

  • User Thoughts Memory

    sickn33/agentic-awesome-skills

    Saves a user's project decisions, rules and preferences into a project-local mdbase so later sessions and other agents can recover the intent.

    47k GitHub starsUsed in 1 repo~2.5k tokens
    Auto-check passed

  • Using LWC Memory and Graphs

    sickn33/agentic-awesome-skills

    Keeps project decisions, research and verified results available across coding-agent sessions through LWC memory, a document Wiki graph and a CodeGraph code index.

    47k GitHub starsUsed in 1 repo~2k tokens
    Auto-check passed

  • Find Complementary Founders

    sickn33/agentic-awesome-skills

    Guides an agent through assessing its own owner for cofounder fit, publishing an approved profile, and ranking complementary profiles other agents published for their owners.

    47k GitHub starsUsed in 1 repo~4.8k tokens
    Auto-check passed

  • Whatsapp Cloud API

    sickn33/agentic-awesome-skills

    Integracao com WhatsApp Business Cloud API (Meta). An agent skill from sickn33/agentic-awesome-skills.

    47k GitHub starsUsed in 2 repos~4.5k tokens
    Auto-check passed

  • Cline Pilot

    sickn33/agentic-awesome-skills

    Acts as a proxy for the Cline CLI, dispatching coding tasks one at a time, monitoring runs by hard evidence, relaying decisions to you and learning per-project preferences.

    47k GitHub starsUsed in 1 repo~4.6k tokens
    Auto-check passed

Categories

Questions about Ab Test Setup

What does Ab Test Setup do?

A skill your agent uses when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules. Ab Test Setup is an agent skill from sickn33/agentic-awesome-skills. Use when designing an A/B or split test: define the hypothesis, control and variants, estimate sample size, verify tracking, and predeclare metrics and stopping rules.

When should I use Ab Test Setup?

Ab Test Setup fits situations like: designing an A/B; split test: define the hypothesis; control and variants; estimate sample size.

How do I install Ab Test Setup in Claude Code?

Run `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -a claude-code`. Or copy the skill folder (skills/ab-test-setup in sickn33/agentic-awesome-skills) into .claude/skills/ab-test-setup in your project. Claude Code loads it when a task matches its description.

How do I install Ab Test Setup in Codex?

Run `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -a codex`. Or copy the skill folder (skills/ab-test-setup in sickn33/agentic-awesome-skills) into .agents/skills/ab-test-setup in your project. Codex loads it when a task matches its description.

Can I use Ab Test Setup in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add sickn33/agentic-awesome-skills --skill ab-test-setup -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/ab-test-setup, .gemini/skills/ab-test-setup, .github/skills/ab-test-setup and .opencode/skills/ab-test-setup in your project.

What does Ab Test Setup need to run?

SKILL.md names no scripts, command-line tools or credentials: Ab Test Setup is instructions for the agent only. Our summary lists: Python 3.

Does Ab Test Setup access the network?

SKILL.md names 1 domain. As links in the text: statsmodels.org. This is read from the text; nothing was executed.

Is Ab Test Setup safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Ab Test Setup use?

Ab Test Setup is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Ab Test Setup use?

About 2.2k tokens (SKILL.md is roughly 8.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Ab Test Setup?

Skills that share tags, products or a category with Ab Test Setup: Ab Test Analyzer (irinabuht12-oss/marketing-skills, 3.8k stars), Define Hypothesis (product-on-purpose/pm-skills, 713 stars), A B Test Design (Owl-Listener/designer-skills, 2.9k stars) and Send Experiment Designer (aaron-he-zhu/aaron-marketing-skills, 2.9k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Ab Test Setup?

sickn33 (a GitHub user) maintains it in sickn33/agentic-awesome-skills, which has 47,304 GitHub stars. The repository holds 1,394 skills in this directory. The repository was last updated on October 6, 2026.

Source: sickn33/agentic-awesome-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.