Survey Design Basics

Survey Design Basics Skill

Surveys fail at design time, invisibly: the leading question ("How much do you love our new feature?"), the double-barrel ("Is the product fast and reliable?" — which one?), the scale with no honest exit (forced positivity), and the twenty-minute questionnaire that only the delighted and the furious complete. By the time results arrive, the damage is unfixable — the data measures the questionnaire, not the population. Design discipline is cheap and front-loaded: neutral wording, one question per question, scales with real options, brutal length editing, a pilot, and the analysis plan written before launch — because a question you don't know how you'll analyze is a question you shouldn't ask.

What This Skill Produces

• The question set — each question bias-checked and single-barreled, mapped to the decision it informs
• The scale choices — response options with the reasoning (and the don't-know/NA exits that keep answers honest)
• The length edit — the cut list, with the completion-rate logic
• The pilot + analysis plan — five test-takers before launch, and the how-each-question-gets-analyzed table written first

Required Inputs

Ask for these if not provided:

• The decision the survey feeds — what will be done differently based on results; questions that inform no decision get cut first (kpi-tracker-design so-what logic)
• The audience and reach method — who gets it, how, and the response-rate reality (the selection caveat gets written into the analysis plan now, not discovered later)
• The draft questions, if any — existing drafts get the bias audit; the classic sins are findable and fixable
• Prior interview themes — surveys size what interviews surfaced (interview-synthesis hands off here); a survey inventing its own hypotheses mid-questionnaire does both jobs badly

Framework: The Design Rules

1. Neutral wording or measured applause: every question checked for leading language ("how much do you love" → "how would you rate"), loaded framing, and social-desirability pull (people report the virtuous answer — anonymity and neutral framing are the mitigations). The test: could a respondent tell which answer you're hoping for?
2. One barrel per question: "fast and reliable" splits into two; "satisfied with price and support" splits into two — every "and" in a question is a fork respondents resolve invisibly, corrupting both halves.
3. Scales with honest exits: balanced options (as many negative as positive), a genuine midpoint where neutrality is real, and don't-know / not-applicable where respondents might legitimately not know — forced answers are fabricated data with a UI. Consistent scale direction throughout (flipping positive-left to positive-right mid-survey harvests inattention, not insight).
4. Length is a completion-rate decision: every question costs respondents; the audit asks each one "which decision do you inform?" and cuts the merely-interesting. Target minutes stated honestly up front; the nice-to-know questions die so the need-to-know ones get answered by more than the furious-and-delighted.
5. Pilot, then the pre-launch analysis plan: five real-ish people take it aloud (confusions found here cost nothing; found in results, everything) — and the analysis table (question → how it's cut → what result triggers what) exists before launch. It catches unanalyzable questions, pre-commits interpretations (guarding against results-fishing), and makes the results memo a fill-in exercise.