Diff-Driven Smoke Tests

Smoke Test — CI-Oriented Validation via Skyvern Browser Tools

Read the diff, classify changes, start the app, and run targeted smoke tests via Skyvern browser tools with Chrome DevTools MCP as fallback.

<!-- NOTE: .agents/skills/smoke-test/SKILL.md is the repository canonical source.
     Keep the synchronized bundled copy in sync with it:
     skyvern/cli/skills/smoke-test/SKILL.md
     Steps 1-4 are copied from .agents/skills/qa/SKILL.md.
     If you fix bugs in /qa's diff-reading, classification, or app startup,
     mirror those fixes here. -->

You changed code. This skill reads the diff, generates targeted smoke tests, and runs each one via Skyvern browser tools - navigate, act, validate, screenshot. If Skyvern tools are unavailable (no Skyvern MCP), fall back to Chrome DevTools MCP, which connects to a real Chrome instance and supports authenticated sessions. It is /qa's CI companion: same diff-reading, same classification, same app startup, formatted for CI output and PR comments with screenshot evidence on every test.

Quick Start

/smoke-test                              # Diff-driven, auto-detect everything
/smoke-test https://staging.example.com  # Explicit app URL
/smoke-test -- focus on the settings page
/smoke-test --pr 11488                   # Target a specific PR

How It Works

1. Read git diff (reused from /qa)
2. Classify changes → identify testable surfaces (reused from /qa)
3. Choose validation strategy (reused from /qa)
4. Pick browser backend (Chrome DevTools MCP or Skyvern)
5. Handle auth (connect to authenticated session or bypass)
6. Start the app if needed (reused from /qa)
7. Generate 3-8 smoke test cases as action sequences (happy paths only)
8. Run each test via browser tools: navigate → act → validate → screenshot
9. Collect results with screenshot evidence
10. Report with embedded screenshots
11. Post to PR with images

Step 1: Understand the Changes

Get the diff

# What files changed?
git diff --name-only HEAD~1     # vs last commit (if changes are committed)
git diff --name-only            # vs working tree (if uncommitted)

# Full diff for context
git diff HEAD~1                 # or git diff for uncommitted

Pick whichever diff has content. If both are empty, there is nothing diff-driven to test.

Read the changed files

Read the full contents of every changed file that affects behavior:

• Frontend files: .tsx, .jsx, .ts, .js, .css, .html
• Backend/API files: routes, controllers, request/response schemas, serializers, handlers
• Backend-internal files: services, workers, business logic, validators, data-layer code
• Tests that changed alongside the implementation

Look for:

• route paths and page entry points
• component names, visible text, forms, buttons, error states
• API endpoints, request params, response fields, auth requirements
• validation logic, branching behavior, feature flags, empty states
• tests that describe the expected behavior

Step 2: Classify the Diff

Use these rules:

• If both frontend/browser and backend changed, treat it as Mixed.
• If only backend internals changed, do not invent unrelated browser tests or random API calls.
• If a backend change might affect the public contract, inspect routes, schemas, and tests before choosing backend-internal.
• If the diff is mostly documentation or comments, keep testing lightweight and report that no behavioral validation was warranted.

Step 3: Choose the Validation Strategy

Frontend/browser mode

Run smoke tests via Skyvern browser tools against the dev server. Validate the specific UI changes plus 1-2 adjacent regression checks.

Backend API mode

Use the repo's documented local startup and auth instructions, start the backend if needed, identify the changed endpoint(s), and run smoke tests that validate the changed contract.

Backend-internal mode

Run the repo's fast verification commands first, then targeted unit/integration/scenario tests for the changed logic. Only run smoke tests if the change affects exposed behavior.

Mixed mode

Validate the backend first, then run frontend smoke tests against the flow that depends on it. If the backend contract is broken, frontend results are not trustworthy.

Step 3b: Pick Browser Backend

Try Skyvern browser tools first. Fall back to Chrome DevTools MCP if Skyvern MCP is unavailable.

Skyvern browser tools (primary)

Use skyvern_browser_session_create, skyvern_navigate, skyvern_act, skyvern_validate, skyvern_screenshot. Works in CI and locally. Create a local=true session to reach localhost.

Chrome DevTools MCP (fallback)

If Skyvern MCP tools are not available, invoke the /chrome-devtools skill to learn the tool API. Key tools:

• list_pages / new_page / navigate_page - page management
• take_screenshot - capture viewport (save to a path within workspace roots)
• take_snapshot - get page structure with element uids for interaction
• click / fill / press_key - interact with elements by uid
• evaluate_script - run JS for health gates and data extraction
• wait_for - wait for content to load

Chrome DevTools MCP connects to a real Chrome instance with the user's profile, so authenticated sessions carry over - no separate auth step needed if the user is already logged in. Useful when the app requires auth that Skyvern sessions can't provide.

Step 3c: Handle Auth

Check whether the app requires authentication:

1. Chrome DevTools MCP with existing session: Call list_pages. If the user already has the app open and authenticated, navigate_page inherits their session. No auth step needed.
2. User needs to log in: Call new_page or navigate_page to the app URL. If a login page appears, tell the user: "Chrome is open to the login page. Please authenticate, then tell me when you're done." Wait for confirmation before proceeding.
3. Auth bypass: If the repo has a local auth-bypass mechanism (env var or dev branch), use it. Check the project's dev setup docs or CLAUDE.md for details.
4. CI / headless: Use Skyvern browser tools with local=true against an unauthenticated dev server, or test only public-facing pages.

Step 4: Start the App

If the user provided a URL argument, skip startup and use that URL directly.

Otherwise, auto-detect common local ports:

5173, 3000, 3001, 8080, 8000, 4200

If none respond, start the most direct repo-documented local command for the changed surface. If the diff needs both frontend and backend running together and the repo provides a combined frontend/backend dev script, prefer that. Only ask the user to start something manually if the repo has no documented command or startup fails.

If the repo requires background processes, start them in the background and keep notes on how you did it.

Step 5: Generate Smoke Test Cases

For each testable surface identified in Steps 2-3, generate a smoke test case as a numbered action sequence using Skyvern browser tools.

Guidelines

• 3-8 tests per PR (stay narrow — test what changed, not everything)
• Happy paths only (smoke level, not deep QA)
• Each test should answer: "does the changed thing still work?"
• For frontend: navigate to the page, interact with the changed element, verify it works
• For backend API: navigate to a page that exercises the API, verify the response
• For mixed: backend-dependent flows first, then frontend that depends on them

Example test cases

Test: Settings save button works after CSS refactor
1. skyvern_navigate(url="http://localhost:5173/settings")
2. skyvern_act(prompt="Fill Company Name with 'Test Corp', click Save")
3. skyvern_validate(prompt="Success toast visible, no error state")
4. skyvern_screenshot()

Test: Dashboard loads after route refactor
1. skyvern_navigate(url="http://localhost:5173/dashboard")
2. skyvern_validate(prompt="Dashboard content visible, no loading spinner stuck")
3. skyvern_screenshot()

Test: Login redirect works
1. skyvern_navigate(url="http://localhost:5173/protected-page")
2. skyvern_validate(prompt="Redirected to login page or auth prompt shown")
3. skyvern_screenshot()

Step 6: Run Tests via Browser Tools

Option A: Skyvern browser tools (primary)

For localhost URLs, create a local browser session:

skyvern_browser_session_create(local=true, timeout=15)

For publicly reachable URLs, create a cloud session instead:

skyvern_browser_session_create(timeout=15)

For each test case, run its action sequence. Every test starts with skyvern_navigate:

skyvern_navigate(url="http://localhost:5173/settings")

CRITICAL: Always include the url parameter in every skyvern_navigate call. Never omit it and rely on the current page - this prevents test-to-test state bleed.

After navigation, run the health gate to catch broken pages early:

In extension mode, use this health gate. Do not call skyvern_evaluate. Before each skyvern_navigate to a route under test, call skyvern_get_errors(clear=True), skyvern_console_messages(clear=True), skyvern_handle_dialog(clear=True), and skyvern_network_requests(clear=True); discard what they return.

skyvern_tab_list()
skyvern_get_errors()
skyvern_console_messages(level="error")
skyvern_get_html(selector="body")
skyvern_find(by="role", value="alert")
skyvern_find(by="role", value="dialog")
skyvern_handle_dialog()

PASS requires the active tab URL to match the expected route, with no unexpected login redirect. The body must contain the expected page content, not only scripts or an empty app container. Require no unexpected error text, visible alerts, visible dialogs, JavaScript errors, console errors, or JavaScript dialog events. skyvern_handle_dialog reads dialog history; JavaScript dialogs are auto-dismissed by default. If a call fails or evidence is incomplete, record FAIL and stop this test. Do not treat missing evidence as PASS.

Outside extension mode, use this health gate:

skyvern_evaluate(expression="(() => {
  const errors = [];
  const body = document.body?.innerText || '';
  if (body.includes('Something went wrong')) errors.push('error_message');
  if (body.includes('Cannot read properties')) errors.push('js_error_in_ui');
  if (/\\bundefined\\b/.test(body) && !/\\bif\\b|\\btypeof\\b|\\bdocument|tutorial|example/i.test(body) && body.length < 5000) errors.push('undefined_text');
  if (body.includes('connection refused')) errors.push('connection_refused');
  if (/sign.?in|log.?in|auth/i.test(window.location.pathname)) errors.push('auth_redirect');
  if (document.querySelector('[role=\"alert\"]')) errors.push('alert_element');
  if (!document.querySelector('main, [role=\"main\"], nav, header, h1, h2, [class*=\"layout\" i], [class*=\"page\" i], [class*=\"app\" i]'))
    errors.push('blank_page');
  return JSON.stringify({ pass: errors.length === 0, errors });
})()")

If the health gate fails, mark the test as FAIL and move on. Otherwise, continue with the test's action steps:

skyvern_act(prompt="Fill Company Name with 'Test Corp', click Save")
skyvern_validate(prompt="Success toast visible, no error state")
skyvern_screenshot()