Debug Integration Test

Debugging Sculptor Integration Tests

Use this skill when an integration test fails and you need to find the root cause.

Common Failure Modes

1. Playwright TimeoutError on Disabled/Missing Element

Symptom: Test fails with playwright._impl._errors.TimeoutError: Locator.click: Timeout 30000ms exceeded and the call log shows the element was found but remained disabled or not visible.

Example error output:

E  playwright._impl._errors.TimeoutError: Locator.click: Timeout 30000ms exceeded.
E  Call log:
E    - waiting for get_by_test_id("TASK_STARTER").get_by_test_id("BRANCH_SELECTOR")
E      - locator resolved to <button disabled dir="ltr" type="button" ...>
E    - attempting click action
E      2 × waiting for element to be visible, enabled and stable
E        - element is not enabled
E      - retrying click action
E      ...
E      49 × waiting for element to be visible, enabled and stable
E        - element is not enabled

Root cause: The UI element exists in the DOM but never reaches the expected state. Common reasons:

• A prerequisite API call failed or never returned (e.g., branch list not fetched)
• The frontend is waiting on data that the backend never provided
• A timing issue where the test interacts with the UI before initialization completes

How to investigate:

1. Look at the Playwright call log in the traceback — it tells you the exact element state (element is not enabled, element is not visible, etc.)
2. Check the backend API logs in the test output for errors or missing API calls
3. View the failure screenshot (test-failed-1.png) — in CI this is always present and shows the page state at the moment of failure. See "Playwright Test Artifacts" below for how to find it. This is often enough to diagnose the issue.
4. Inspect the Playwright trace if the screenshot and logs aren't sufficient. The trace contains full DOM snapshots at each action, so you can:
   • Read frame-snapshot entries in trace.trace to see the complete DOM tree at the moment of timeout, including all data-testid attributes and element states
   • Read trace.network to check whether the expected API calls were made before the timeout
5. Compare what API calls were made (visible as HTTP log lines in Captured stdout and in trace.network) against what the frontend needs

2. "File not found" in Electron Window (DEV_ELECTRON mode)

Symptom: The Electron window shows {"detail":"File not found: "} as plain JSON text instead of the React app. Tests using the sculptor_instance_ shared fixture fail with ERROR at setup and the traceback shows expect(home_page.get_sidebar_toggle_button()).to_be_visible() timing out at resources.py. The test log shows "GET / HTTP/1.1" 404.

Root cause: In DEV_ELECTRON mode (the default), the React frontend is served by the Vite dev server, not the backend. The backend's catch-all route @APP.get("/{filename:path}") in app.py tries to serve static files from frontend-dist/ or frontend/dist/, but those directories don't contain built files in dev mode. If code navigates the Playwright page to the backend URL (e.g., page.goto(http://127.0.0.1:{backend_port})), the backend returns a 404 with the "File not found" JSON detail.

How to identify:

• Search the log for "GET / HTTP/1.1" 404 — confirms the backend received a root request and returned 404
• Search for ERR_ABORTED (-3) — Electron's error when a page navigation is aborted
• The error occurs during fixture setup, not during the test itself

How to fix: Ensure that test infrastructure code does not navigate to the backend URL in DEV_ELECTRON mode. The Electron window already loads the React app from the Vite dev server. For the shared sculptor_instance_ fixture, _get_or_create_shared_instance in resources.py should skip navigate_to_frontend and just wrap the existing page in PlaywrightHomePage.

Key file: sculptor/sculptor/testing/resources.py — the _get_or_create_shared_instance function.

3. FakeClaude Command Parsing Errors

Symptom: Task enters ERROR state immediately after creation, or FakeClaude returns an unexpected response. The test log may contain JSON parsing errors from the FakeClaude command handler.

Root cause: The fake_claude: command string has malformed JSON — e.g., missing backticks, unescaped quotes, or incorrect nesting.

How to identify: Search the test log for json.JSONDecodeError or FakeClaude error messages.

How to fix: Check the command string formatting. FakeClaude commands must follow this format:

fake_claude:<command> `<json_args>`

The JSON must be wrapped in backticks. Use multiline strings for complex JSON to avoid formatting mistakes.

Playwright Test Artifacts

When a test fails, Playwright records several artifacts that are invaluable for debugging. The available artifacts depend on the flags passed to pytest:

• test-failed-1.png — a full-page screenshot taken at the moment of failure (--screenshot=only-on-failure). Start here — it's small, fast to view, and often enough to diagnose UI issues.
• trace.zip — a full trace containing the action timeline, DOM snapshots at each step, network requests, console logs, and screencast frames (--tracing=retain-on-failure). Use this when the screenshot alone isn't enough.
• video.webm — a screen recording of the entire test run (--video=retain-on-failure).

pytest.ini only configures --tracing=retain-on-failure. The CI runner adds --screenshot=only-on-failure and --video=retain-on-failure. When running locally, add these flags yourself if you want the screenshot and video.

Where to find artifacts

Locally — after a failed test run, look in:

sculptor/test-results/<test-file-stem>-<test-name>-<browser>/

For example, a failure in test_task_page_chatting.py::test_starting_text produces:

sculptor/test-results/test-task-page-chatting-test-starting-text-chromium/trace.zip

In CI — artifacts are uploaded as workflow artifacts. To download them:

1. Download the full artifacts archive for the failed run (find the run ID with gh run list):

   gh run download <RUN_ID> --dir /tmp/ci-artifacts/

   or download a single named artifact: gh run download <RUN_ID> -n <artifact-name> --dir /tmp/ci-artifacts/.

2. The download contains test artifacts organized under test-results/<test-dir>/, where <test-dir> is derived from the test file and test name. Depending on the test configuration, each directory may include failure screenshots (e.g. test-failed-1.png), Playwright traces, and video recordings. For example:

   test-results/tests-integration-frontend-test-ask-user-question-py-test-ask-user-question-dismiss/test-failed-1.png

3. List the downloaded files to find artifacts for a specific test:

   find /tmp/ci-artifacts -path '*test-results*' | grep <test-name>

What's inside a trace zip

A trace.zip is a standard ZIP archive. Unzip it to access the contents:

unzip trace.zip -d /tmp/trace-contents/

Key files inside: