QA Changes

QA Changes

Validate pull request changes by actually running the code — not just reading it. The goal is to verify that new behavior works as the PR claims, existing behavior is not broken, and the repository remains healthy after the change.

The bar is high: test the way a thorough human QA engineer would. If the PR changes a web UI, spin up the server and verify it in a real browser. If it changes a CLI, run the CLI with real inputs. Do not settle for "the tests pass" — actually use the software.

Core Methodology

QA proceeds in four phases. Complete each phase in order. If a phase fails, report the failure and stop.

Phase 1: Understand the Change

Read the PR diff, title, and description. Identify the goal of this PR — this is the single most important thing to understand before proceeding. A PR might fix a bug, add a feature, refactor code, improve performance, update documentation, or something else entirely. Check:

1. The PR description "Why" / "Summary" section — what is the author trying to accomplish?
2. Linked issues — if the PR references an issue, read it. But note: the PR may address the issue differently than expected, or only partially. The PR description is the real specification for what this PR intends to deliver.
3. The PR title — often summarizes the intent (e.g., "fix: X not working when Y", "feat: add Z capability", "refactor: consolidate duplicated X logic").

Then classify every changed file:

• New feature: User-visible behavior that did not exist before.
• Bug fix: Corrects existing behavior to match intended behavior.
• Refactor: Restructuring that should not change external behavior.
• Configuration / CI / docs: Non-functional changes.

For each change, identify the entry point — the concrete way a user would interact with it (CLI command, API endpoint, UI page, function call). This drives what to exercise in Phase 3.

Finally, form a clear hypothesis: "This PR should [achieve stated goal] by [approach taken in the diff]." Phase 3 will test that hypothesis.

Phase 2: Set Up the Environment

Bootstrap the repository so the project builds and runs successfully.

1. Read the repo's bootstrap instructions. Check AGENTS.md, README.md, Makefile, package.json, pyproject.toml, Cargo.toml, or equivalent. Always prefer the project's own documented setup commands.
2. Install dependencies. Use the project's dependency manager (uv sync, npm install, pip install -r requirements.txt, bundle install, cargo build, etc.).
3. Build the project if a build step is required (compile, transpile, bundle).
4. Note CI status. Glance at the PR's CI checks and note whether they pass or fail. Do NOT re-run the test suite yourself — that is CI's job, not yours. Your job starts in Phase 3.

If setup fails, report the failure with the exact error output and stop.

Phase 3: Exercise the Changed Behavior

This is the most important phase. Actually use the software the way a real user would to verify the change works as the PR claims. This is what distinguishes QA from CI (which runs tests) and code review (which reads code).

Do NOT:

• Run the test suite (pytest, npm test, cargo test, etc.) — that is CI's job.
• Analyze code by reading files and commenting on style, structure, or logic — that is code review's job.
• Run linters, formatters, type checkers, or pre-commit hooks — that is CI's job.

DO:

• Run the actual application, CLI, or server and interact with it as a user would.
• Make real HTTP requests, run real commands, open real browser pages.
• Always attempt real execution first. Running --help, --dry-run, or --version is NOT functional verification — it only proves argument parsing works. If real execution fails due to missing credentials, external services, or environment constraints, report what you tried and what could not be verified. Do not substitute --help output for evidence the software works.
• Reproduce bugs and verify fixes end-to-end.
• Test user-facing behavior that automated tests cannot or do not cover.

Start by verifying the PR achieves its stated goal. Use the hypothesis from Phase 1. For example:

• If the PR claims to "fix crash when X is empty", reproduce the crash scenario and confirm it no longer occurs.
• If the PR claims to "add support for Y", actually use Y end-to-end and confirm it works.
• If the PR claims to "add a new dashboard page", navigate to the page and verify it renders and functions correctly.
• If the PR claims to "add a new CLI flag", run the CLI with that flag and verify the output.

"Tests pass" is not a QA finding. The question is: does the software actually do what the PR says it does?

For frontend / UI changes:

• Start the development server.
• Use a real browser (via Playwright, browser automation tools, or the built-in browser) to navigate to the affected pages.
• Verify the visual change renders correctly. Take screenshots as evidence.
• Test user interactions (clicks, form submissions, navigation).
• Try at least one edge case (empty state, long text, missing data).
• Then capture visual evidence and attach it to the PR (see next step).

Capture visual evidence (frontend PRs only):

If the PR has frontend work, run the affected screens from the branch's final state and record what a reviewer would otherwise have to imagine from the diff. The goal: the change can be reviewed by observation, not by reading code.

1. Screenshot each relevant state of every affected screen - whichever apply: empty, loading, error, and populated. Skip states the screen genuinely cannot reach.
2. Record the key interaction end to end - the main flow the PR changes, driven the way a user would. Prefer a GIF: GitHub renders a GIF inline from an image link, but only links a video file.
3. Where behavior changes, capture before/after - the same state or interaction on the base branch and on the PR branch, so the delta is visible.
4. Attach it to the report. Embed the images and GIF in the QA report so they render inline, and label each one (screen and state, or before/after). Do not push commits to the PR branch to host media: that changes the head you are verifying, and the shipped QA workflow has read-only access to repository contents. If you cannot upload media where the report is posted, save the files where the run keeps its artifacts, say where to find them, and describe in text what each capture shows. In the qa-changes GitHub Action that is $GITHUB_WORKSPACE/output/, the parent of the PR checkout you work in, which the action uploads as the openhands-qa-changes-logs artifact; a relative output/ inside the checkout is not uploaded.

Keep it proportional: capture the screens the PR actually touches, not the whole app. If you cannot render a screen (missing data, an unreachable state, no browser), say so in the report rather than faking it.

For CLI changes:

• Run the CLI command with realistic arguments. Capture stdout and stderr.
• Verify the output matches the PR's claimed behavior.
• Try at least one edge case (invalid input, missing flags, empty input).

For API / backend changes:

• Start the server.
• Make actual HTTP requests (curl, httpie, or a test client) to affected endpoints.
• Verify response status codes, response bodies, and side effects (database writes, file creation).
• Test error cases (bad input, missing auth, not found).

For bug fixes — use a before/after comparison:

1. Reproduce the bug without the fix. Check out the base branch (or revert the PR's changes) and run a concrete command or code path that triggers the reported failure. Show the exact command and its output.
2. Interpret the baseline result. Explain what the output means — e.g., "This confirms the bug exists: the resolver cannot find the package because the lockfile's cutoff date is too old."
3. Apply the PR's changes. Check out the PR branch, apply the patch, or set the environment variable — whatever the fix entails.
4. Re-run the same verification. Run the same command or exercise the same code path with the fix in place. Show the exact command and its output.
5. Interpret the result. Explain what the new output means — e.g., "The resolver now finds the package, confirming the fix works."
6. Check for side effects. Confirm the fix does not break related functionality.

For library / SDK changes:

• Write a short script that imports and calls the changed functions.
• Verify the return values and behavior match the PR's claims.
• Test edge cases the PR author may have missed.

For refactors:

• If the refactor touches a critical or user-facing path, manually exercise that path to confirm behavior is unchanged.
• For pure internal refactors where CI passes and no user-facing path is affected, Phase 2's CI check is sufficient.

For configuration / CI / docs:

• Validate syntax (YAML lint, JSON parse, markdown render).
• If it is a build change, confirm the build still succeeds.
• For doc changes, confirm the documentation renders correctly if a preview is available.

Always show your work with a before/after narrative. For every verification, the report must include: (a) the exact command you ran, (b) the actual output you observed, and (c) your interpretation of that output. For bug fixes and behavioral changes, demonstrate BOTH the broken/old state AND the fixed/new state so the reviewer can see the delta. Present this evidence inside collapsible <details> blocks — the core deliverable is the verdict and summary, not raw logs.