Uplift Ttsim CI

Stage 4: Run Every Candidate in Isolation

Keep the noop edit active. For each architecture, run the corresponding candidates.txt serially. Quote each complete node ID:

timeout --signal=TERM --kill-after=10s 180s \
  env PYTHONUNBUFFERED=1 \
  /opt/ttmlir-toolchain/venv/bin/python -m pytest -svvv "$node_id" \
  --sys-desc "$TT_METAL_SIMULATOR_HOME/ttrt-artifacts/system_desc.ttsys" \
  --path "$TT_METAL_SIMULATOR_HOME/pytest_artifacts_candidates" \
  --tb=short -rs

Write one log per node plus a TSV/JSON manifest containing architecture, node ID, elapsed time, exit code, and result. Continue after nonzero exits and print progress after every node. WH and BH loops may run concurrently, but never group multiple node IDs into one pytest invocation. Keep -s enabled so pytest capture cannot swallow the last TTSim error when the process exits.

Classify results from both exit status and the pytest terminal summary:

• passed: pytest reports the node passed. This is the only automatic unskip evidence.
• failed: pytest prints a normal failure/error summary.
• aborted: the process ends without a pytest terminal summary, even if its exit code is only 1.
• timed_out: the 180-second timeout expires.
• skipped, xfailed, or xpassed: record separately; none proves a clean simulator pass.

For an empty or unclear abort log, rerun that exact node with PYTHONUNBUFFERED=1 -svvv. Do not rerun a group.

After the matrix is complete:

• Remove a simulator mark only when every architecture to which it applies passed.
• Narrow mixed results to the smallest dtype, shape, op, or architecture scope. Single-chip WH is n150; single-chip BH is p150.
• If a mark on one parametrization axis mixes passing and failing combinations from another axis, restructure the parametrization so the mark names the exact combinations; do not retain the broad axis-level skip.
• A config list is AND'd; separate groups are OR'd. skip_config(["n150", "sim"]) means WH simulator only, while SkipIf("n150", "sim") means n150 or any simulator.
• Preserve unrelated skips. For example, if only sim is obsolete in SkipIf("ttnn", "emitc", "emitpy", "sim"), keep SkipIf("ttnn", "emitc", "emitpy").

If behavior differs from the previous TTSim version, run the smallest failing node against both versions with the same tt-mlir checkout and corresponding system descriptors.

Stage 5: Restore Triage State

Restore _get_current_environment() to return "sim" before any final validation. Inspect the diff rather than using a destructive checkout, and verify that conftest.py has no triage-only change. Keep the candidate manifests as validation artifacts, not source changes.

Stage 6: Run the Full Regression Sweep

This stage is mandatory and happens after skip updates and noop restoration.

Run every file in TEST_FILES, one file per pytest process, on WH and BH. Do not use one aggregate pytest command: an abort would hide the remaining files. Continue after failures. The two architecture loops may run in parallel.

timeout --signal=TERM --kill-after=15s 3600s \
  env PYTHONUNBUFFERED=1 \
  /opt/ttmlir-toolchain/venv/bin/python -m pytest -vv "$test_file" \
  --sys-desc "$TT_METAL_SIMULATOR_HOME/ttrt-artifacts/system_desc.ttsys" \
  --path "$TT_METAL_SIMULATOR_HOME/pytest_artifacts_regression" \
  --tb=short -rs

Some files may take more than 30 minutes, but an individual node should not take more than about three minutes. Monitor each live -vv log at least every three minutes. If progress stops, terminate the file run, isolate the last reported node with the 180-second candidate command, update the smallest necessary skip, and rerun that file. A generous file timeout is not a substitute for progress monitoring.

Record exactly one final status per file per architecture:

• passed: pytest exits 0 with a terminal summary; expected skips/xfails are allowed.
• failed: pytest reports normal failures.
• aborted: no pytest terminal summary, regardless of exit code.
• timed_out: the file timeout expires.

Any skip change made during regression requires a fresh run of that file on both architectures. Do not finish with an untriaged failed, aborted, or timed out file.

Stage 7: Final Verification and Handoff

1. Run focused WH and BH pytest for every file whose skips changed.
2. Verify all workflow-listed files have a final WH and BH status.
3. Syntax-check .github/workflows/call-test-ttsim.yml with available local tooling.
4. Run git diff --check and inspect git status and the complete diff.
5. Confirm conftest.py is restored and generated simulator assets/results are not in the source diff.
6. Summarize:
   • release tag and verified asset names;
   • descriptor YAML and generated .ttsys used per architecture;
   • candidate counts and result totals per architecture;
   • simulator skip removals, additions, and narrowings;
   • per-file regression status for WH and BH;
   • focused reruns and workflow validation;
   • any explicit remaining coverage gap.