Evidence Grading

Evidence Grading Skill

"We have evidence" covers everything from a randomized experiment to the CEO's seatmate on a flight — and decisions made on ungraded evidence inherit the confusion. The grading discipline: inventory what actually supports the claim, place each item on the business-evidence hierarchy (what it is, not how confident it feels), test the graded weight against the decision's stakes (a reversible pilot needs less than a one-way rebrand — decision-journal logic), and when evidence falls short, name the cheapest upgrade — because the answer to weak evidence is usually a better test, not a braver bet.

What This Skill Produces

• The evidence inventory — everything supporting (and contradicting) the claim, each item graded on the hierarchy
• The weight assessment — what the graded pile actually supports, including the mixed-signals honest read
• The sufficiency verdict — enough for this decision's stakes / not yet — with the stakes analysis shown
• The upgrade path — the cheapest next evidence that would move the verdict ("a 2-week holdout test settles this for $0")

Required Inputs

Ask for these if not provided:

• The claim and the decision riding on it — "users want X" feeding a backlog item vs. feeding a repositioning are different sufficiency bars; the decision's reversibility and cost set the bar
• The evidence, itemized — every piece: the data pull, the survey, the five customer quotes, the competitor's move, the expert's opinion — including the inconvenient items (an inventory that omits contradicting evidence is advocacy)
• The evidence's provenance — n, selection, dates, who collected it and with what incentive (source-triangulation supplies the externals; internal evidence has incentives too)

Framework: The Grading Rules

1. The hierarchy, applied without sentiment: experiments/A-B tests (causal) > usage/behavioral data (what people do, correlational) > surveys (what they say, at scale — survey-design-basics quality adjusts the grade) > interviews (rich, small-n — interview-synthesis counts matter) > anecdotes (existence proofs only) > expert opinion (informed priors) > internal conviction (a hypothesis, not evidence). Each item gets its rung and its quality-within-rung — a leading survey grades below honest interviews.
2. Direction and independence both count: the inventory includes contradicting evidence at its own grade, and echo-checks the supporting pile (three anecdotes traceable to one loud customer are one anecdote). The weight is the net, honestly netted.
3. Anecdotes prove existence, never prevalence: "a customer asked for X" establishes that the need exists somewhere — it cannot establish how common; the classic grading error is prevalence conclusions from existence evidence, and it's the error this skill most often catches.
4. Sufficiency is relative to stakes: the verdict tests weight against the decision — reversible-and-cheap decisions legitimately run on interview-grade evidence (the pilot is the experiment); irreversible-and-expensive ones demand behavioral or experimental grade. "Weak evidence" isn't a verdict; "weak for this bet" is.
5. The upgrade path is the constructive ending: when insufficient, name the cheapest test that would change the verdict — the holdout, the fake-door, the survey that sizes the interview theme, the pilot-with-metrics. Ranked by cost-to-confidence ratio; the skill's product is often not "no" but "this $0 two-week test first."