Exploratory Testing

Assisted Exploration (LLM as Companion, Not Replacement)

You can run a session with an LLM as oracle and idea-generator while keeping critical-thinking ownership. This maps cleanly onto the testing-vs-checking distinction in Bach & Bolton's Taking Testing Seriously (Wiley 2025): the LLM can help with checking (does this match a known reference?) but the testing — the human judgment about what to explore and what counts as a problem — stays yours. Done well, an LLM expands your charter coverage; done badly, it replaces your judgment with confident-sounding hallucination.

How to use an LLM during a session:

• As an idea generator before the session. Paste the charter and ask for 10 edge cases the heuristics might miss. Pick 3 to actually try. Discard the rest — most will be generic or invented.
• As an oracle for "is this correct?" mid-session. When you find unexpected behavior, ask the agent to look up the spec / API / standard. Never trust the answer without verifying against the source it cites.
• As a fact-checker on findings, not a writer of bug reports. You write the bug; the LLM reviews for clarity. The reverse — LLM writes, you review — produces template-shaped reports that lose the specific details a human noticed.
• For coverage gap suggestions during debrief. "Given these notes, what charter should I run next?"

The productivity-paradox warning: AI tooling can make tester output look faster while quietly hollowing out the critical thinking that produced the value. (Michael Bolton has argued this line in DevelopSense talks and posts; treat it as a working principle, not a cited finding.) If your debrief notes start sounding like an LLM wrote them, the LLM is now driving — stop and run the next session unassisted.

What never to delegate to an LLM:

• Choosing what to explore. The charter must come from your understanding of risk and stakeholder concerns.
• Deciding whether something is a bug. "The model says it looks fine" is not a debrief.
• Writing the testing story. Specific, situated detail is the point of exploratory testing — generic LLM prose is the opposite.

For testing AI features themselves (not just using AI to test), see ai-system-testing.

Tester Roles in Modern Teams

Useful vocabulary for staffing and self-positioning conversations. These are common industry archetypes, not a formal taxonomy:

• Embedded testers — testers fully embedded inside delivery teams, contributing to development conversations end-to-end rather than acting as a separate gate. The most common model on cross-functional teams.
• Specialist testers — deep skills in a domain (security, accessibility, performance) called in across teams.
• Coach testers — senior testers who teach craft (heuristics, charter writing, exploratory thinking) to developers and junior testers; rarely test end-to-end themselves.

If your org is moving toward embedded testers, exploratory testing is one of the highest-leverage skills to demonstrate — it is hard for developers to pick up without coaching, and it is where the testing mindset shows up most clearly.

For background on the testing-vs-checking distinction and AI's role, see "What Is Testing? A Conversation with Bach and Bolton" (DevelopSense, Feb 2026): https://developsense.com/blog/2026/02/what-is-testing-a-conversation-with-james-bach-and-michael-bolton

Anti-Patterns

Unchartered Exploration

Exploring without a charter. "I will just poke around and see what I find" produces inconsistent results, is not repeatable, and cannot be meaningfully debriefed. Always write a charter, even if it is one sentence.

Session Too Long

Running 3-hour exploration sessions. Bug-finding effectiveness drops sharply after 90 minutes. Fatigue causes testers to miss issues and stop following leads. Break long testing efforts into multiple 60-90 minute sessions with breaks between them.

No Notes During Session

Relying on memory to reconstruct what happened. By the time the session ends, half the observations are forgotten and the rest are vague. Take notes in real time using the template above.

Exploring Only the Happy Path

Using exploratory testing only to verify that things work. This duplicates what automated tests already cover. Exploratory testing's strength is finding problems in paths nobody thought to script. Use heuristics to push into uncomfortable territory.

No Conversion to Automation

Finding the same bug manually every release because nobody wrote an automated test for it. Every reproducible bug found through exploration should become an automated regression test. The exploration-to-automation pipeline must be active.

Treating Exploration as "Not Real Testing"

Viewing exploratory testing as less rigorous than scripted testing. SBTM with charters, session logs, and debriefs produces documented, accountable testing. The documentation format is different from test scripts, but the rigor is equal.

Verification

Prove the session produced real, accountable artifacts — not just a feeling that you "tested it" — smallest check first:

• Each charter is a real charter: every session has a written Explore [target] with [resources] to discover [information] line where the target is more specific than "the app." A session with no charter line is unchartered exploration, not SBTM.
• The session log has timestamps and tags: open the log and confirm observations carry a time column and a tag (BUG, QUESTION, IDEA, RISK, NOTE). An untimed, untagged wall of prose cannot be debriefed.
• Every filed bug traces back to a charter and reproduces: for each bug ID, follow its reproduction steps in a clean environment and confirm it still happens. A bug you can't reproduce is a note, not a filed defect.
• Converted regression tests actually run red-then-green: for any exploratory finding promoted to automation, run the new test against the buggy build (npx playwright test path/to/spec should fail) and against the fix (should pass). A regression test that was never seen failing is not proven to guard the bug.
• Coverage is stated honestly: the debrief marks each charter area covered, partial, or unexplored. "100% covered" with unexplored areas listed below it is the gap to catch.

Done When

• Session charters are written for each target area, each following the "Explore [target] with [resources] to discover [information]" pattern
• All planned sessions have been executed and debriefed using the debrief template, with each charter area marked covered, partial, or unexplored
• Every bug found during sessions is logged with a reference to the originating charter and reproduction steps
• Session logs exist with time-stamped observations tagged as BUG, QUESTION, IDEA, RISK, or NOTE
• A findings summary captures total session count, bugs filed (by severity), test ideas identified, and follow-up sessions scheduled or explicitly deferred

Reference Files (in references/)

• session-templates.md — Charter examples, environment-prep checklist, session-flow timings, debrief template, and the note-taking session-log format.
• heuristics-and-automation.md — Boundary/state/error/"what if" heuristic test-idea banks, the exploration-to-automation pipeline diagram, and a worked Playwright regression example.

Related Skills

• agentic-browser-testing -- The automated cousin: a browser agent explores the app from a natural-language goal with no script. Use it for unattended exploratory smoke; use exploratory-testing for human, charter-driven SBTM sessions and bug hunting.
• playwright-automation -- Where stabilized exploratory findings graduate into maintained, deterministic regression tests.
• test-planning -- Sprint test plans allocate time for exploratory sessions and reference charters.
• risk-based-testing -- Risk assessment identifies which areas deserve exploratory attention.
• test-reliability -- Flaky or unreliable areas identified through exploration feed into test reliability improvements.
• qa-metrics -- Track exploratory session counts, bug discovery rates, and charter coverage as QA metrics.
• qa-project-context -- The project context file identifies known risk areas and previous bug clusters that guide charter writing.