Stress Test

Skill: Stress Test

Type: Standalone — reviews any analysis plan for methodological flaws

---

Purpose

Takes an analysis plan, investigation design, or analytical approach and pressure-tests it for hidden flaws — wrong baselines, survivorship bias, missing segments, uncontrolled confounds, and absent kill criteria. Acts as a "senior data scientist code review" for your analytical thinking.

This skill is standalone — it works on any analysis plan, whether produced by /analysis-design, written by the user, or pulled from an existing document.

---

When to Use

• Before committing a week to executing an analysis plan
• Before presenting an analysis design to stakeholders
• When you've written an analysis brief and want a gut-check
• When reviewing someone else's analytical approach
• After an analysis came back with unexpected results (was the design flawed?)

---

Inputs

---

The 7-Point Stress Test

Check data availability first. A methodologically perfect plan that needs data the dataset lacks is unexecutable, so this check precedes the seven checkpoints:

1. Read the active dataset schema (.knowledge/datasets/{active}/schema.md)
2. Check if the plan's required fields/tables/dimensions exist in the dataset
3. If ANY required data is missing → HALT, skip checkpoints 1-6, jump straight to checkpoint 7, assign FAIL verdict with BLOCKER status, provide F grade, and stop
4. If all required data exists → proceed with checkpoints 1-7 in order

Common data blockers to check for:

• Plan requires device/platform segmentation → check if device, platform, or user_agent fields exist
• Plan requires funnel analysis → check if event-level tracking exists (page views, clicks, sessions)
• Plan requires traffic source attribution → check if source, medium, campaign, or referrer fields exist
• Plan requires geographic segmentation → check if country, region, city, or geo fields exist
• Plan assumes conversion rate → check if both numerator (orders, signups) AND denominator (sessions, page views) data exist

---

After confirming data availability, review the plan against each checkpoint. For each one, assign a verdict: PASS, WARNING, or FAIL.

1. Hypothesis Clarity

Check: Is there a specific, testable hypothesis? Or is this a fishing expedition?

Fix if FAIL: "State what you expect to find AND what would prove you wrong."

2. Baseline Validity

Check: Is the comparison group / baseline appropriate?

Common traps to check:

• Comparing to a period with a holiday, major campaign, or outage
• Before/after with no control group
• Comparing cohorts of different sizes or compositions
• Using "last month" when the metric is seasonal

Fix if FAIL: "Identify 2-3 alternative baselines and explain why each is valid or invalid."

3. Survivorship Bias

Check: Does the analysis only look at entities that "survived" to be measured?

Common traps:

• "Average revenue per user" that only counts users who made a purchase
• "Onboarding completion rate" that only counts users who started onboarding
• "Feature satisfaction" survey that only reaches users who didn't churn
• Retention analysis that excludes users who churned before the measurement window

Fix if FAIL: "Explicitly define the denominator. Who is EXCLUDED from this analysis and why?"

4. Segment Coverage

Check: Will the analysis check for segment-level differences that could reverse the overall finding?

Key segments to always check:

• Device (mobile/desktop/tablet)
• Platform (iOS/Android/web)
• User tenure (new/returning/power user)
• Geography (if multi-market)
• Traffic source / acquisition channel
• Product category / feature area
• Customer tier / plan type

Fix if FAIL: "Add segment breakdowns for every categorical dimension with >2 values. Flag any reversal."