Test Design

Designing Matft tests

Matft promises "behaves like NumPy". Most of the bugs found so far were not in the main path but at the edges: integer wraparound, zero-length dimensions corrupting the heap, slice views reading past their data, complex imaginary parts copied from the real part, vDSP behaving differently on x86_64, NaN dropped by a kernel. A test that only checks f([1, 2, 3]) in .Float catches none of these. This skill turns "write tests" into a systematic pass over the viewpoints where Matft actually breaks, with every expected value coming from numpy so the test cannot encode a misunderstanding.

TDD is required in this repository (CLAUDE.md): write the tests, watch them fail, then implement.

Workflow

1. Pin down the numpy specification

Before writing anything, run the numpy counterpart in Python and read its docs for the target function: signature and defaults, the output dtype rule, the output shape for each axis/keepdims, what it does with NaN, inf, empty input, negative axes, out-of-range parameters. Also note where Matft deliberately differs (see "Matft conventions" below) so you don't report them as bugs, and check Sources/ for the existing signature if the function exists.

Use the venv from python/README.md (.venv/bin/python). If it does not exist, create it as the README says.

2. Make a test plan over the viewpoints

Read references/viewpoints.md and go through every viewpoint for the target. For each one, decide cover (list the concrete cases) or N/A (with a one-line reason, e.g. "no axis parameter"). Write the plan as a short table in your reply before writing code. The table is what makes the coverage reviewable: a missing row is visible, an unexplained gap is not.

Size the plan to the change: a new public function gets the full pass; a one-line bug fix gets the failing case plus the neighbouring viewpoints that the same code path touches (e.g. a fix in a reduction kernel → axes, layouts, dtypes).

3. Generate the expected values with numpy

Expected values are never computed by hand or copied from Matft's own output — that would make the test agree with the bug. Choose one of:

• Generated test file (default for anything with more than a handful of cases). Add cases to an existing generator (python/gen_numpy_gaps_coverage.py for numpy-level functions, gen_fft_audio_coverage.py, gen_image_coverage.py), or create python/gen_<area>_coverage.py from assets/gen_template.py. Each case writes the Swift expression and the numpy expression side by side, so a reviewer can check them against each other. The output file starts with // Generated by ... Do not edit by hand. Register a new generator in the table of python/README.md. Details: references/generator.md.
• Hand-written XCTest for things a generator expresses badly: in-place mutation and aliasing, thrown errors, view identity, or a single regression case. Still compute the numbers in Python and paste them with the numpy expression as a comment (// numpy: np.diff(a, n=4, axis=1).shape -> (3, 0)).

Compare arrays with XCTAssertClose (= np.testing.assert_allclose, NaN matches NaN, inf must match exactly) and pass checkType: true — a wrong output dtype is a numpy mismatch too. For exact Int / Bool results use rtol: 0, atol: 0. Avoid XCTAssertEqual(MfArray, MfArray): == ignores the mftype, treats floats within 1e-5 as equal, wraps integers into the type (UInt8 index 299 "equals" 43) and never matches NaN, so it hides exactly the bugs this skill is looking for. XCTAssertEqual is fine for shapes and Swift scalars. Tolerances: Float rtol/atol ≈ 1e-5..1e-6, Double 1e-10..1e-12; loosen only with a comment explaining why (e.g. accumulated FFT error).