Agent skill

Ab Testing

by Cesarjoquin in Cesarjoquin/Marketing-Skills

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.

MITAuto-check passedMarketing & SEO

Install Ab Testing

skills CLI
$ npx skills add Cesarjoquin/Marketing-Skills --skill ab-testing -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install Cesarjoquin/Marketing-Skills ab-testing --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/Cesarjoquin/Marketing-Skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/ab-testing .claude/skills/ab-testing && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
ab-testing
GitHub stars
199
Used in
2 other repos
Token cost
~2.8k tokens
SKILL.md length
1,108 words
Files
4 (incl. references)
Skills in repo
12
Repo updated
First seen
Licence
MIT

At a glance

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.

  • Works in 4 steps: Start with a Hypothesis → Test One Thing → Statistical Rigor → …
  • Implement an A/B test
  • SKILL.md covers Initial Assessment, Core Principles, Hypothesis Framework and Test Types, plus 7 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Ab Testing is an agent skill from Cesarjoquin/Marketing-Skills. When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program," or "experiment…

Its SKILL.md is about 2.8k tokens, which your agent loads only when the skill is triggered. The skill folder holds 5 other files, including reference files (for example `evals/evals.json`, `references/sample-size-guide.md` and `references/test-templates.md`).

It sits in Marketing & SEO, covering A/B testing. The repository describes itself as: AI agent Marketing skills for Claude Code and AI agents. CRO, copywriting, SEO, analytics, ai agent, and growth engineering, ai agent. The licence is MIT.

When your agent uses it

  • Implement an A/B test
  • Build a growth experimentation program
  • The user mentions A/B test
  • Test this change

Example prompts

  • “A/B test,”
  • “split test,”
  • “experiment,”
  • “/ab-testing”

Workflow steps

4 steps, taken from the step headings in SKILL.md.

  1. Start with a Hypothesis
  2. Test One Thing
  3. Statistical Rigor
  4. Measure What Matters

What it can do on your machine

Read from SKILL.md and the folder at commit 23b887f. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md.

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • evanmiller.org
    • optimizely.com

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Ab Testing loads about 2.8k tokens when it runs, and up to ~6.3k if it reads all its reference files. Until then it costs about 199 tokens; SKILL.md has 1,108 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~199
When it runs · the whole SKILL.md, loaded when a task matches
~2.8k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~6.3k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Cesarjoquin/Marketing-Skills at commit 23b887f, republished under its MIT licence (© Cesarjoquin). 1,108 words, ~2,788 tokens.

Download SKILL.mdSave it as .claude/skills/ab-testing/SKILL.md (or your agent's skills folder). This skill also uses 3 other files; get the full folder from GitHub.
name
ab-testing
description
When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," "how long should I run this test," "growth experiments," "experiment velocity," "experiment backlog," "ICE score," "experimentation program," or "experiment playbook." Use this whenever someone is comparing two approaches and wants to measure which performs better, or when they want to build a systematic experimentation practice. For tracking implementation, see analytics. For page-level conversion optimization, see cro.
metadata.version
2.0.0

A/B Test Setup

You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.

Initial Assessment

Check for product marketing context first: If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.

Before designing a test, understand:

  1. Test Context - What are you trying to improve? What change are you considering?
  2. Current State - Baseline conversion rate? Current traffic volume?
  3. Constraints - Technical complexity? Timeline? Tools available?

Core Principles

1. Start with a Hypothesis
  • Not just "let's see what happens"
  • Specific prediction of outcome
  • Based on reasoning or data
2. Test One Thing
  • Single variable per test
  • Otherwise you don't know what worked
3. Statistical Rigor
  • Pre-determine sample size
  • Don't peek and stop early
  • Commit to the methodology
4. Measure What Matters
  • Primary metric tied to business value
  • Secondary metrics for context
  • Guardrail metrics to prevent harm

Hypothesis Framework

Structure
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Example

Weak: "Changing the button color might increase clicks."

Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."


Test Types

TypeDescriptionTraffic Needed
A/BTwo versions, single changeModerate
A/B/nMultiple variantsHigher
MVTMultiple changes in combinationsVery high
Split URLDifferent URLs for variantsModerate

Sample Size

Quick Reference
Baseline10% Lift20% Lift50% Lift
1%150k/variant39k/variant6k/variant
3%47k/variant12k/variant2k/variant
5%27k/variant7k/variant1.2k/variant
10%12k/variant3k/variant550/variant

Calculators:

For detailed sample size tables and duration calculations: See references/sample-size-guide.md


Metrics Selection

Primary Metric
  • Single metric that matters most
  • Directly tied to hypothesis
  • What you'll use to call the test
Secondary Metrics
  • Support primary metric interpretation
  • Explain why/how the change worked
Guardrail Metrics
  • Things that shouldn't get worse
  • Stop test if significantly negative
Example: Pricing Page Test
  • Primary: Plan selection rate
  • Secondary: Time on page, plan distribution
  • Guardrail: Support tickets, refund rate

Designing Variants

What to Vary
CategoryExamples
Headlines/CopyMessage angle, value prop, specificity, tone
Visual DesignLayout, color, images, hierarchy
CTAButton copy, size, placement, number
ContentInformation included, order, amount, social proof
Best Practices
  • Single, meaningful change
  • Bold enough to make a difference
  • True to the hypothesis

Traffic Allocation

ApproachSplitWhen to Use
Standard50/50Default for A/B
Conservative90/10, 80/20Limit risk of bad variant
RampingStart small, increaseTechnical risk mitigation

Considerations:

  • Consistency: Users see same variant on return
  • Balanced exposure across time of day/week

Implementation

Client-Side
  • JavaScript modifies page after load
  • Quick to implement, can cause flicker
  • Tools: PostHog, Optimizely, VWO
Server-Side
  • Variant determined before render
  • No flicker, requires dev work
  • Tools: PostHog, LaunchDarkly, Split

Running the Test

Pre-Launch Checklist
  • Hypothesis documented
  • Primary metric defined
  • Sample size calculated
  • Variants implemented correctly
  • Tracking verified
  • QA completed on all variants
During the Test

DO:

  • Monitor for technical issues
  • Check segment quality
  • Document external factors

Avoid:

  • Peek at results and stop early
  • Make changes to variants
  • Add traffic from new sources
The Peeking Problem

Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.


Analyzing Results

Statistical Significance
  • 95% confidence = p-value < 0.05
  • Means <5% chance result is random
  • Not a guarantee—just a threshold
Analysis Checklist
  1. Reach sample size? If not, result is preliminary
  2. Statistically significant? Check confidence intervals
  3. Effect size meaningful? Compare to MDE, project impact
  4. Secondary metrics consistent? Support the primary?
  5. Guardrail concerns? Anything get worse?
  6. Segment differences? Mobile vs. desktop? New vs. returning?
Interpreting Results
ResultConclusion
Significant winnerImplement variant
Significant loserKeep control, learn why
No significant differenceNeed more traffic or bolder test
Mixed signalsDig deeper, maybe segment

Show full SKILL.md (453 more words)Show less

Documentation

Document every test with:

  • Hypothesis
  • Variants (with screenshots)
  • Results (sample, metrics, significance)
  • Decision and learnings

For templates: See references/test-templates.md


Growth Experimentation Program

Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.

The Experiment Loop
1. Generate hypotheses (from data, research, competitors, customer feedback)
2. Prioritize with ICE scoring
3. Design and run the test
4. Analyze results with statistical rigor
5. Promote winners to a playbook
6. Generate new hypotheses from learnings
→ Repeat
Hypothesis Generation

Feed your experiment backlog from multiple sources:

SourceWhat to Look For
AnalyticsDrop-off points, low-converting pages, underperforming segments
Customer researchPain points, confusion, unmet expectations
Competitor analysisFeatures, messaging, or UX patterns they use that you don't
Support ticketsRecurring questions or complaints about conversion flows
Heatmaps/recordingsWhere users hesitate, rage-click, or abandon
Past experiments"Significant loser" tests often reveal new angles to try
ICE Prioritization

Score each hypothesis 1-10 on three dimensions:

DimensionQuestion
ImpactIf this works, how much will it move the primary metric?
ConfidenceHow sure are we this will work? (Based on data, not gut.)
EaseHow fast and cheap can we ship and measure this?

ICE Score = (Impact + Confidence + Ease) / 3

Run highest-scoring experiments first. Re-score monthly as context changes.

Experiment Velocity

Track your experimentation rate as a leading indicator of growth:

MetricTarget
Experiments launched per month4-8 for most teams
Win rate20-30% is common for mature programs (sustained higher rates may indicate conservative hypotheses)
Average test duration2-4 weeks
Backlog depth20+ hypotheses queued
Cumulative liftCompound gains from all winners
The Experiment Playbook

When a test wins, don't just implement it — document the pattern:

## [Experiment Name]
**Date**: [date]
**Hypothesis**: [the hypothesis]
**Sample size**: [n per variant]
**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]

Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.

Experiment Cadence

Weekly (30 min): Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.

Bi-weekly: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.

Monthly (1 hour): Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.

Quarterly: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?


Common Mistakes

Test Design
  • Testing too small a change (undetectable)
  • Testing too many things (can't isolate)
  • No clear hypothesis
Execution
  • Stopping early
  • Changing things mid-test
  • Not checking implementation
Analysis
  • Ignoring confidence intervals
  • Cherry-picking segments
  • Over-interpreting inconclusive results

Task-Specific Questions

  1. What's your current conversion rate?
  2. How much traffic does this page get?
  3. What change are you considering and why?
  4. What's the smallest improvement worth detecting?
  5. What tools do you have for testing?
  6. Have you tested this area before?

  • cro: For generating test ideas based on CRO principles
  • analytics: For setting up test measurement
  • copywriting: For creating variant copy

© Cesarjoquin, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 3 other files (references) in skills/ab-testing of Cesarjoquin/Marketing-Skills.

  • SKILL.md
  • evals/evals.json
  • references/sample-size-guide.md
  • references/test-templates.md

Open the folder on GitHubat commit 23b887f

Used in 2 other repositories

We found 4 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 2 other GitHub owners. This page covers the copy in Cesarjoquin/Marketing-Skills, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Ab Testing next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Ab Testing compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Ab Testing this skillCesarjoquin/Marketing-Skills1992 repos~2.8kAutomated safety check: PassMIT
AnalyticsNexus-JPF/note-companion8696 repos~2.2kAutomated safety check: PassMIT
Ab Test Setupfreekmurze/dotfiles1k15 repos~1.8kAutomated safety check: PassNone
Ad Test Designeraaron-he-zhu/aaron-marketing-skills2.9k2 repos~2.8kAutomated safety check: PassApache-2.0
Meta Tags Optimizernowork-studio/notfair-plugin3.9k1 repos~2.7kAutomated safety check: PassMIT
Ab Test Analyzeririnabuht12-oss/marketing-skills3.8k—~1.4kAutomated safety check: PassNone

Similar skills

  • Analytics

    Nexus-JPF/note-companion

    When the user wants to set up, improve, or audit analytics tracking and measurement.

    869 GitHub starsUsed in 6 repos~2.2k tokens
    Marketing & SEOAuto-check passed
  • Ab Test Setup

    freekmurze/dotfiles

    When the user wants to plan, design, or implement an A/B test or experiment.

    1k GitHub starsUsed in 15 repos~1.8k tokens
    Marketing & SEOAuto-check passed
  • Ad Test Designer

    aaron-he-zhu/aaron-marketing-skills

    A skill your agent uses when the user asks to "design an A/B test", "set up a creative/landing test", "run an incrementality test", or "is this result statistically and practically material?"…

    2.9k GitHub starsUsed in 2 repos~2.8k tokens
    Marketing & SEOAuto-check passed
  • Meta Tags Optimizer

    nowork-studio/notfair-plugin

    Writes and improves title tags, meta descriptions, Open Graph and Twitter card tags for click-through, with character counts and A/B test variants.

    3.9k GitHub starsUsed in 1 repo~2.7k tokens
    Marketing & SEOAuto-check passed
  • Ab Test Analyzer

    irinabuht12-oss/marketing-skills

    Statistical significance calculator for A/B test results with sample size requirements, segment breakdowns, and hypothesis generation.

    3.8k GitHub stars~1.4k tokensUpdated 13 days ago
    Marketing & SEOAuto-check passed
  • Ab Test Store Listing

    appeeky/aso-skills

    When the user wants to A/B test App Store product page elements to improve conversion rate.

    2.1k GitHub stars~1.8k tokensUpdated yesterday
    Marketing & SEOAuto-check passed

More from Cesarjoquin/Marketing-Skills

All 12 skills in this repo
  • Ads

    Cesarjoquin/Marketing-Skills

    When the user wants help with paid advertising campaigns on Google Ads, Meta (Facebook/Instagram), LinkedIn, Twitter/X, or other ad platforms.

    199 GitHub starsUsed in 1 repo~3.5k tokens
    Auto-check passed
  • Emails

    Cesarjoquin/Marketing-Skills

    When the user wants to create or optimize an email sequence, drip campaign, automated email flow, or lifecycle email program.

    199 GitHub starsUsed in 1 repo~2.1k tokens
    Auto-check passed
  • Prospecting

    Cesarjoquin/Marketing-Skills

    When the user wants to find, qualify, and build a list of prospects to reach out to — across B2B SaaS, general B2B, or local small businesses.

    199 GitHub starsUsed in 1 repo~3.8k tokens
    Auto-check passed
  • Referrals

    Cesarjoquin/Marketing-Skills

    When the user wants to create, optimize, or analyze a referral program, affiliate program, or word-of-mouth strategy.

    199 GitHub starsUsed in 1 repo~1.9k tokens
    Auto-check passed
  • Aso

    Cesarjoquin/Marketing-Skills

    When the user wants to audit or optimize an App Store or Google Play listing.

    199 GitHub stars~3.9k tokensUpdated 23 days ago
    Auto-check passed
  • Paywalls

    Cesarjoquin/Marketing-Skills

    When the user wants to create or optimize in-app paywalls, upgrade screens, upsell modals, or feature gates.

    199 GitHub stars~1.4k tokensUpdated 23 days ago
    Auto-check passed

Categories

Questions about Ab Testing

What does Ab Testing do?

When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program. Ab Testing is an agent skill from Cesarjoquin/Marketing-Skills. When the user wants to plan, design, or implement an A/B test or experiment, or build a growth experimentation program.

When should I use Ab Testing?

Ab Testing fits situations like: implement an A/B test; build a growth experimentation program; the user mentions A/B test; test this change.

How do I install Ab Testing in Claude Code?

Run `npx skills add Cesarjoquin/Marketing-Skills --skill ab-testing -a claude-code`. Or copy the skill folder (skills/ab-testing in Cesarjoquin/Marketing-Skills) into .claude/skills/ab-testing in your project. Claude Code loads it when a task matches its description.

How do I install Ab Testing in Codex?

Run `npx skills add Cesarjoquin/Marketing-Skills --skill ab-testing -a codex`. Or copy the skill folder (skills/ab-testing in Cesarjoquin/Marketing-Skills) into .agents/skills/ab-testing in your project. Codex loads it when a task matches its description.

Can I use Ab Testing in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add Cesarjoquin/Marketing-Skills --skill ab-testing -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/ab-testing, .gemini/skills/ab-testing, .github/skills/ab-testing and .opencode/skills/ab-testing in your project.

What does Ab Testing need to run?

SKILL.md names no scripts, command-line tools or credentials: Ab Testing is instructions for the agent only.

Does Ab Testing access the network?

SKILL.md names 2 domains. As links in the text: evanmiller.org and optimizely.com. This is read from the text; nothing was executed.

Is Ab Testing safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Ab Testing use?

Ab Testing is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Ab Testing use?

About 2.8k tokens (SKILL.md is roughly 11k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 3.5k tokens, read only when the agent opens those files.

What are the alternatives to Ab Testing?

Skills that share tags, products or a category with Ab Testing: Analytics (Nexus-JPF/note-companion, 869 stars), Ab Test Setup (freekmurze/dotfiles, 1k stars), Ad Test Designer (aaron-he-zhu/aaron-marketing-skills, 2.9k stars) and Meta Tags Optimizer (nowork-studio/notfair-plugin, 3.9k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Ab Testing?

Cesarjoquin (a GitHub user) maintains it in Cesarjoquin/Marketing-Skills, which has 199 GitHub stars. The repository holds 12 skills in this directory. The repository was last updated on September 14, 2026.

Source: Cesarjoquin/Marketing-Skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.