Fix Reproduce

Reproduce — Verify the Bug Exists

Runs a test and determines whether the bug reproduces. Uses a three-stage approach: nightly wheel first (fast), source build at CI commit second (precise), CI environment alignment third (last resort).

The orchestrator decides what to do with the output — this skill only reports the result.

Inputs

• reproducer_command — the sequence of shell commands that triggers the failure. Any of the forms below is valid; Stage 1's "Reproducer forms" section routes execution:

  1. A pytest node id or pytest ... invocation: pytest -v test/xpu/test_ops.py::TestFooXPU::test_bar_xpu_float32
  2. A python -c "..." snippet or a single python script.py line.
  3. A multi-line shell block (as is common in issue bodies): env setup, git clone, pip install, followed by the actual failing command.

  Missing or unrunnable inputs return NO_REPRODUCER up-front — see "## Preflight" below. Non-NO_REPRODUCER verdicts come from the Stage 1/2/3 execution flow.

  Providers (set by the orchestrator, not this skill):

  • Issue body — extracted by issue-triage from the reproducer section (via issue-handler).
  • CI failure log — the failing pytest node id from the nightly CI report, passed through as a batch sub-item.

• stage — which reproduction path to run. Default auto.

  • auto — run the full three-stage fallback chain (nightly → source_build → ci_env). Used by orchestrators that need a definitive verdict.
  • nightly — only run Stage 1 (nightly wheel). PASS returns NOT_REPRODUCED(checked_stages=[nightly]) immediately; do NOT fall through to source build or ci_env. Cheapest option — suitable for a fast "does this still reproduce on latest nightly?" answer.

• ci_commit — upstream commit hash from the CI report. Only used as a fallback base when origin/main fails to build (optional; ignored when stage=nightly).

• pytorch_dir — path to a local PyTorch checkout (optional). Used whenever Prepare determines needs_tree=yes (any pytest form with a repo-relative path) as well as by Stage 2's source build. If absent, clone to <torch-xpu-ops-repo-root>/agent_space_xpu/pytorch/ (agent_space_xpu/ is the gitignored scratch dir at the torch-xpu-ops repo root — see the containing repo's AGENTS.md).

• ci_repo — which CI to align against in Stage 3: pytorch or torch-xpu-ops. Optional; when absent, Stage 3 infers from the reproducer path (see "Determine ci_repo" in Stage 3).

Preflight

NO_REPRODUCER is a pre-execution verdict — the skill decides that there is nothing to run before touching any stage. Every other verdict (REPRODUCED / NOT_REPRODUCED / CANNOT_VERIFY) comes from Stage 1/2/3 execution.

reproducer_command present? If missing or empty: NO_REPRODUCER(reason=no_command). Stop.

If the input is a stack trace without a command, the orchestrator should not call this skill in the first place (that is issue-triage's reproduction_missing=yes case); if it slips through, this check catches it.

The pytest collected 0 items check happens in Prepare below, after the source tree it needs to run against is in place.

Prepare

Some reproducer forms need a pytorch source tree even at Stage 1 (nightly wheel path) — either because the test file lives inside pytorch/test/ or because it lives under torch-xpu-ops/test/xpu/ and imports common test utilities via sys.path.append("../../../../test/functorch") relative paths that only resolve from <pytorch_dir>/third_party/torch-xpu-ops/test/xpu/.

When to prepare

Set needs_tree from the reproducer form:

If needs_tree=no, skip this section and go to Stage 1.

Get the pytorch tree

If pytorch_dir was provided as input: git -C $pytorch_dir fetch origin.

If not provided, clone into the torch-xpu-ops repo's gitignored scratch dir. Resolve the path explicitly rather than relying on cwd:

XPU_OPS_ROOT=$(git -C <path-to-torch-xpu-ops-checkout> rev-parse --show-toplevel)
pytorch_dir="$XPU_OPS_ROOT/agent_space_xpu/pytorch"
if [[ ! -d "$pytorch_dir/.git" ]]; then
  git clone --filter=blob:none https://github.com/pytorch/pytorch.git "$pytorch_dir"
fi
git -C "$pytorch_dir" fetch origin
git -C "$pytorch_dir" checkout --detach origin/main
git -C "$pytorch_dir" submodule update --init --recursive

The tree is not built here — Stage 1 uses the nightly wheel for the runtime; the source tree only supplies test files and support modules. Stage 2 reuses the same tree and builds it there.

torch-xpu-ops test path

If the reproducer targets test/xpu/..., make the working torch-xpu-ops tree available at $pytorch_dir/third_party/torch-xpu-ops. The build's dev-override recipe (symlink or replace-clone) applies; see xpu-build-pytorch. From here on, Stage 1's cwd for pytest is $pytorch_dir/third_party/torch-xpu-ops/test/xpu/.

Collect-only check (pytest form)

Regardless of needs_tree, if reproducer_command matches the pytest form (starts with pytest, python -m pytest, or is a bare pytest node id), run:

pytest --collect-only <node_id>

Cwd:

• needs_tree=yes → from the tree just prepared (for test/xpu/... targets, that's $pytorch_dir/third_party/torch-xpu-ops/test/xpu/)
• needs_tree=no → from any non-pytorch directory (the reproducer's absolute path resolves on its own)

Output shows collected 0 items? → NO_REPRODUCER(reason=collected_zero). Stop. Do not fall through to Stage 2 — the source tree is the same across stages, so a collect-miss at Stage 1 will miss at 2 and 3 too.

Non-pytest forms have no equivalent pre-execution check.

Stage 1: Nightly Wheel (fast path)

Most failures reproduce here. Start here before doing anything heavier.

Reproducer forms

Three forms; each dispatches differently in "Run test" below:

• Pytest form — reproducer_command starts with pytest, python -m pytest, or is a bare pytest node id (.../test_foo.py::TestBar::test_baz). The collect-only check (see "## Prepare" above) has already run.
• Python one-liner / single-script form — python -c "..." or python path/to/script.py.
• Shell-block form — a multi-line block copied out of an issue body: env setup + git clone + pip install + the failing command. Split it into setup steps (everything before the failing command) and the reproduce step (the last command that exercises the failing path). Only the reproduce step's outcome determines the verdict; setup-step failures return CANNOT_VERIFY(stage=<current>, blocker=<the failing setup step>).

The Working directory and Use the test's own assertion rules below apply to all three forms.

Install

Always reproduce against the latest available XPU nightly. Do not reuse a stale wheel from a previous session — a bug may already be fixed in a newer nightly, and re-verifying an old wheel produces misleading REPRODUCED results.

# Query available versions (informational — pip install --upgrade below
# will pick a resolvable one, which may lag the newest entry here by a
# day when the index metadata refreshes before all wheels land).
pip3 index versions torch --pre \
  --index-url https://download.pytorch.org/whl/nightly/xpu
pip3 install --pre --upgrade torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/nightly/xpu

Post-install, check that torch, torchvision, torchaudio are all from the same day — pip's resolver can leave a mixed set (either torch older than the auxiliary wheels, or the reverse). If they diverge, uninstall all three and reinstall together:

pip3 uninstall -y torch torchvision torchaudio
pip3 install --pre torch torchvision torchaudio \
  --index-url https://download.pytorch.org/whl/nightly/xpu

Record the exact wheel version used (python -c "import torch; print(torch.__version__)") in the reproduce output and in any issue comment, so downstream stages and re-verifications know which nightly was tested.

Working directory

Do NOT run the nightly-wheel reproducer with cwd inside any pytorch source checkout. Python resolves import torch against the local torch/ package before site-packages, so it will load the in-tree torch/_C.so built at whatever revision that tree happens to be — typically stale relative to the installed wheel — and fail with ImportError: undefined symbol: .... Either cd $(mktemp -d) (or any non-pytorch dir) before running, or invoke the reproducer with an absolute path from outside the tree.

torch-xpu-ops test invocation

Applies when the reproducer targets a test under torch-xpu-ops/test/xpu/. Prepare has already ensured the pytorch tree exists at $pytorch_dir with the working torch-xpu-ops tree at $pytorch_dir/third_party/torch-xpu-ops. Invoke the reproducer from $pytorch_dir/third_party/torch-xpu-ops/test/xpu/ — the relative sys.path.append("../../../../test/functorch") in those tests only resolves from that cwd.

The pytorch tree does NOT need to be built for the nightly-wheel path — the wheel provides the runtime, the source tree only supplies test files and support modules.

Use the test's own assertion

When writing a standalone reproducer for a TestCase.assertEqual failure, use the test's own assertion. Do NOT substitute torch.allclose, torch.equal, or bare == — they have different (usually stricter) default tolerances and will manufacture false positives.

If the failure log says AssertionError: Tensor-likes are not close, the assertion is torch.testing._comparison.assert_close, which has dtype-specific defaults (bf16: rtol=0.016, atol=1e-5). Reproduce through assert_close or via TestCase.assertEqual:

import sys; sys.path.insert(0, "<pytorch>/test")
from torch._dynamo.test_case import TestCase   # or the base class the failing test uses

class T(TestCase):
    def test_x(self, device):
        ...
        self.assertEqual(out_ref, out)

T().test_x(device='xpu')

Run test

Run according to the reproducer form matched in "Reproducer forms":

• Pytest form: run the pytest invocation. Result interpretation:

  • FAILED → REPRODUCED.

  • all skipped by @skipIfXpu → the marker is likely hiding the actual failure. Temporarily remove it in place, re-run once to check what happens without the skip, then revert the file so no change escapes this skill:

    # Remove @skipIfXpu from the target test file(s), then:
    pytest <node_id>
    # Regardless of outcome, revert:
    git checkout <test_file>

    If the re-run FAILs → REPRODUCED (the skip was hiding it). If it PASSes → treat as PASSED per the Decision table. Only return CANNOT_VERIFY when the skip is environmental (not an XPU marker, e.g. @skipIf(not has_cuda) shielding an unavailable dep).

  • xfailed → treat as FAILED (REPRODUCED).

  • PASSED → per the Decision table below.

• Python one-liner / shell block: run the command (or the reproduce step extracted from the shell block, per "Reproducer forms"). Result interpretation:

  • Exit code non-zero and output matches the failure pattern named in the issue (traceback, error message, or specific assertion) → REPRODUCED.
  • Exit code zero → PASSED (per the Decision table below).
  • Exit code non-zero but cause is unrelated (missing dependency surfacing inside the reproduce step, permission error, missing device) → CANNOT_VERIFY(stage=nightly, blocker=<...>). Do NOT report REPRODUCED on a setup or infra failure.

  all skipped / xfailed do not apply to these forms — they are pytest-specific concepts.

Decision

Stage 2: Source Build at origin/main

Nightly passing is not conclusive — it may lag behind CI. Build from origin/main to verify. Even when the failure came from a specific CI commit, we only consider fixes on top of origin/main — downstream stages branch off it.

Prepare pytorch checkout

If Prepare (above) already ran (needs_tree=yes), $pytorch_dir is detached at origin/main with submodules initialized — skip to "Build and run".

Otherwise (Prepare was skipped because needs_tree=no), run Prepare's "Get the pytorch tree" recipe now: resolve $pytorch_dir (from input or $XPU_OPS_ROOT/agent_space_xpu/pytorch), clone if missing, fetch, checkout --detach origin/main, submodule update --init --recursive.

Leave HEAD detached at origin/main at exit (the ci_commit fallback below re-detaches to a different sha; downstream stages branch off whatever this stage settled on).

Build and run

Load the xpu-build-pytorch skill and follow it for the build. Do not hand-roll the build here.

If the origin/main build fails for a reason unrelated to the bug (broken trunk, upstream infra issue, etc.) and ci_commit is available, fall back once:

git -C $pytorch_dir checkout --detach $ci_commit
git -C $pytorch_dir submodule update --init --recursive

Rebuild via xpu-build-pytorch. If this succeeds, proceed with the test on ci_commit and record base=<ci_commit_sha> in the output so the orchestrator branches its fix off the same base. If the fallback build also fails, escalate as CANNOT_VERIFY(blocker=trunk and ci_commit both fail to build) rather than silently reproducing on some other base.

Then run the reproducer following the form-specific rules in Stage 1 "Run test" (pytest interpretation vs python/shell interpretation).

Decision

Applies only when stage=auto — stage=nightly returns at Stage 1.

Stage 3: CI Environment Alignment

Only reached when nightly wheel and source build at origin/main both pass. The failure may be specific to the CI environment: wheels built under CI toolchain, XPU/oneAPI stack pinned to a specific version, environment variables set by CI.