Grill Me QA

Grill Me QA

A guided, systematic interview that challenges every aspect of a QA automation plan or test strategy until all decisions are resolved. Designed for QA Automation Engineers, SDETs, and QA Leads working with AI-assisted testing tools.

Activation: Triggered when the user wants to validate, challenge, or stress-test a testing plan, test strategy, framework choice, test architecture, or AI-testing strategy. Also activated by explicit reference: "use the skill grill-me-qa".

When to Use This Skill

• Before starting a new test automation project or framework
• When adopting AI tools for test generation, healing, or analysis
• Before presenting a test strategy to stakeholders
• When refactoring or migrating an existing test suite
• When evaluating tool/framework trade-offs (Playwright vs Selenium vs Cypress vs k6)
• Before scaling test coverage or reorganizing a regression suite
• When flaky tests are undermining trust in the suite

Trigger Phrases

The skill activates on explicit requests and on implicit intent. Treat these as activation signals in both English and Spanish:

• "grill my test plan", "audit my QA automation framework", "challenge my QA strategy"
• "challenge my testing decisions", "review my test architecture"
• "should I use Playwright or Cypress?", "how should I structure my tests?"
• "my tests are flaky, help me figure out why"
• "migrate from Selenium to Playwright", "is my test pyramid healthy?"
• "how do I set up quality gates?", "why is my test suite so slow?"
• "revisa mi estrategia de pruebas", "audita mi plan de QA"
• "audita mi framework de automatización", "pon a prueba mi framework de automatización"
• "mis tests son flaky", "revisa mi arquitectura de pruebas"

When NOT to Use This Skill

This skill grills strategy and architecture decisions. It is not the right tool for:

• Writing or debugging a single test → use playwright-e2e-testing, api-testing, or the relevant framework skill
• Generic code review of production (non-test) code
• Fixing one isolated failing test (that is debugging, not strategy)

If the request is tactical ("write me a login test", "why does this selector fail"), hand off to the implementation skills instead.

Scope & Exclusions

This skill targets web and API test automation. The five dimensions apply broadly, but tooling recommendations (Dimension 2) and non-functional requirements (Dimension 1) are tuned for web/API contexts.

• Mobile-native (Appium / XCUITest / Espresso), embedded, or hardware-in-the-loop testing: the dimensions still apply, but the framework and NFR recommendations will not map cleanly. Flag this in Phase 1 and adapt the Dimension 2/1.3 answers accordingly rather than recommending web tooling.
• Mobile and contract testing are intentionally out of scope as dedicated concerns — they belong in separate skills.

How Grilling Works

Core Protocol

1. One question at a time. Never ask multiple questions simultaneously. Wait for the user's answer before proceeding. Piling up questions overwhelms and produces shallow answers.
2. Provide a recommended answer for every question, based on industry best practices, ISTQB principles, and the specific context of the project. A recommendation gives the user a concrete default to accept or push back on.
3. Explore before asking. If a question can be answered by examining the codebase, test suite, CI configuration, or existing documentation — explore it instead of asking the user. Respecting the user's time is part of being thorough.
4. Walk the decision tree systematically. Follow the five dimensions in order. Resolve dependencies between decisions one by one before moving to the next branch.
5. Challenge vague answers. If the user's answer is imprecise, follow up to sharpen it. "It depends" is not an answer — force a concrete decision with context.
6. Track decisions in memory. Maintain a running log of resolved decisions, deferred items, and open questions throughout the session. Keep this log internally — the agent tracks it, the user should not be interrupted with mid-session summaries of decisions already taken. Surface the full decision record only at the end in the TSDR.

Adaptive Recommendations

Adapt your recommended answers to the profile identified in Phase 1. The right advice for a solo SDET is wrong for a QA Lead at an enterprise:

• SDET (individual): pragmatic, fast to implement, favors off-the-shelf tooling over bespoke frameworks.
• QA Lead: strategic, enterprise-grade, emphasizes governance, metrics, and team scalability.
• QA Engineer (individual): practical, team-oriented, balances shipping speed with maintainability.

Session Flow

Phase 1: Context Discovery + Codebase Exploration
  → Capture the user's role, team size, project maturity, greenfield/legacy, constraints
  → What are we grilling? (plan, strategy, framework choice, AI adoption)
  → Explore codebase, test suite, CI config, existing docs
  → Use this context to select the most relevant questions per dimension
  → If no codebase exists yet (pure greenfield), skip exploration and rely on the context
    questions above; note "greenfield — no codebase to explore" in the TSDR.

Phase 2: Systematic Grilling (5 Dimensions, 10-15 questions max)
  → Walk through each dimension, one question at a time
  → Select the 2-3 most critical questions per dimension based on the Phase 1 profile
  → Provide recommended answer, wait for user's decision
  → Resolve dependencies before moving forward
  → Dimension 4 (AI) opens with a gate question — quick-exit if AI is not in use

Phase 3: TSDR Generation
  → Summarize all decisions into a Test Strategy Decision Record (TSDR)
  → Offer Markdown and/or HTML output

Question Budget

The whole session should stay within 10-15 questions. A grilling that drags past 15 questions exhausts the user and dilutes the signal. The budget counts only the interrogation questions across the five dimensions — Phase 1 context discovery questions are free and do not count against the limit.

Accounting rules, so the budget actually holds:

• Phase 1 (Context Discovery) is free — role, team size, greenfield/legacy, and constraint questions do not count toward the 10-15.
• 2-3 core questions per dimension, with Dimension 4 (AI) and Dimension 5 carrying up to 4 (ceiling = 3+3+3+4+4 = 17). Aim for 2 per dimension unless the dimension is high-risk for this project; in practice the D4 gate usually trims Dimension 4 to ≤2.
• Dimension 4 (AI) gate validation questions count against the budget (at most 2 when AI is a quick-exit).
• Deep-dives are a trade-off, not an addition: if a dimension runs its full core set, deep-dives are forbidden there unless you cut another dimension down. Treat the 15 ceiling as the target — 16 is only reached when Dimension 4 runs its full core.
• Depth matters more than coverage — a few sharp questions resolve more than a long checklist of generic ones.

The Five Dimensions

Each dimension contains a decision tree with specific interrogation points. Read references/qa-decision-tree.md for the full question bank.

Dimension 1: Test Strategy & Coverage

• What testing pyramid shape fits this project?
• What is the risk-based coverage priority?
• What is the boundary between automated and exploratory testing?
• What non-functional testing is required (performance, security, accessibility)?

Dimension 2: Framework & Tooling

• Why this framework over alternatives? (Playwright / Selenium / Cypress / k6 / other)
• What language and why? (TypeScript / Java / Python / other)
• What design pattern? (Page Object Model / Screenplay / Fluent API / hybrid)
• What reporting and observability tools?

Dimension 3: Test Architecture

• How are tests isolated from each other?
• What is the test data strategy? (seeding, cleanup, factories, fixtures)
• How are environments managed? (local, staging, CI)
• What is the parallelization strategy?
• What is the retry and flakiness policy?

Dimension 4: AI Integration

• Where does AI enter the testing workflow? (generation, healing, analysis, visual, triage)
• How are AI-generated tests validated for correctness?
• What is the human-in-the-loop boundary?
• How are hallucinations and non-deterministic outputs mitigated?
• What is the cost/token budget for AI-assisted testing?
• What AI tools are selected and why?

Gate question: Start Dimension 4 with: "Are you using or planning to use AI tools for testing?" Branch on the answer:

• No → resolve with 1-2 quick validation questions (at most 2, they count against the budget), mark the dimension as Resolved/Not Applicable, and move on. Don't force a full AI interrogation on a team with no AI integration.
• Yes → run the core AI questions (4.1–4.4). In 4.1, probe the current touchpoints, not aspirational ones.
• Ambiguous ("evaluating", "partially", "ad-hoc", "developers use Copilot but nothing is automated") → treat as Yes and run the core questions, but focus 4.1 on the actual current usage. If the answer is "we evaluated it and decided against it," treat that as No and log the rationale.

A non-committal answer is itself a signal worth grilling — never let "maybe" pass without a decision, per Core Protocol rule #5.