Grade Tests

Grade Tests

Assess a curated list of test methods and produce a compact, PR-comment-friendly report. The primary result is one of Pass, Failed, Uncertain, or Not applicable; an A-F quality grade remains secondary diagnostic information. The skill does not discover tests on its own — the caller (typically a PR automation workflow or a human reviewer holding a specific list) provides the tests or a bounded diff to assess.

After Step 0 admits a bounded scope, enforce these grading invariants:

• With production context, load test-gap-analysis by name once before scoring, in per-test-read-only caller context. Read its owned composition reference; do not compute mutation evidence from this grading rubric or run its standalone workflow. Report N/A / unverified only when the dependency, reference, or required context is actually unavailable.
• Any reported mutation inference must use Likely killed (inferred) or Candidate survivor (unverified), even when explained in prose.
• Apply only the rubric below, not extra heuristics such as a duplicate-test penalty. Do not use sibling tests to alter the individual assessment.
• A B quality grade does not require a Failed result; a complete focused test may have no actionable change.

Language-specific guidance: If the caller supplies the matching bundled extension file path, read it directly. Otherwise call test-analysis-extensions to discover available extension files, then read the file matching the target codebase's language and framework (e.g., extensions/dotnet.md, extensions/python.md, extensions/typescript.md, extensions/go.md). You MUST read the relevant extension file before scoring assertions or anti-patterns, because assertion APIs and idiomatic patterns differ significantly across frameworks.

Why a Decision Result Plus Quality Detail

PR reviewers need a simple answer to does this test need follow-up? The four-state result provides that decision; the existing A-F rubric explains its quality and severity.

When to Use

• A PR automation workflow needs to post a decision on the tests introduced or modified in a pull request.
• A reviewer has a specific list of tests (a file, a class, a method list, or a diff hunk) and wants per-test follow-up decisions rather than a suite report.
• A maintainer wants to triage which of N tests in a contribution deserve follow-up improvements, with quality grades for resolved tests.

When Not to Use

• The caller wants a full suite audit or comparative metrics — use test-anti-patterns (pragmatic) or test-smell-detection (formal) and let the test-engineer agent orchestrate its internal quality specialist.
• The caller wants to write new tests — use test-engineer (any language) or writing-mstest-tests (MSTest specifically).
• The caller wants to measure code coverage or CRAP scores — use coverage-analysis or crap-score (.NET only).
• The caller wants to fix issues directly in test code — invoke the appropriate editing skill.
• No specific list of tests is provided. Do not try to grade every test in the workspace; ask the caller for an explicit list or scope.

Inputs

Step 0: Validate the input

Before doing anything else, check that the caller provided one of:

1. An explicit list of test method names, or
2. One or more file paths plus an explicit instruction to grade every test declared in those files (e.g., "grade every test in OrderTests.cs"), or
3. A diff hunk or PR identifier whose changed tests should be graded.

If the request is ambiguous (e.g., "Grade my tests", "Are these tests any good?" with no scope, "Review the test suite"), do not load extensions, do not read files, and do not grade anything. Reply with a short message asking the caller to provide an explicit list / file(s) / diff, and optionally point them at the test-engineer agent or test-anti-patterns skill for full-suite analysis. Stop there.

If a valid bounded scope resolves to zero eligible tests, return Not applicable with a short explanation and no invented rows.

Workflow

Step 1: Detect language and load extension

Identify the target codebase's language and test framework from the file extensions and the test method markers in the provided list. Call the test-analysis-extensions skill unless the caller already supplied the matching bundled extension file path. In either case, read that extension file (e.g., extensions/dotnet.md for MSTest/xUnit/NUnit/TUnit, extensions/python.md for pytest, extensions/typescript.md for Jest/Vitest, extensions/go.md for the standard testing package). If the input contains tests from multiple languages, load each relevant extension and grade each test using its language's conventions.

Step 2: Resolve the test bodies

For each entry in the input list:

1. If the test body is provided inline, use it directly.
2. Otherwise read the file at the given path and locate the method by its fully-qualified name. Capture the full method body, including attributes / decorators / fixtures and any helper code that the test calls.
3. If a requested method cannot be found, record it as Uncertain — method not found with no quality grade and continue. Never invent a body to grade. A missing requested method requires human review; it is not the same as a valid scope containing no tests.

Composition checkpoint: for resolved tests with available production context, load test-gap-analysis now, once for the batch, with per-test-read-only assessment context. Complete its owned reference assessment before Step 3. Do not skip this load just because a body-level weakness already seems obvious; a locally invented mutation explanation is not composition.

Step 3: Assess the claimed behavior and score each resolved test

Keep grading read-only: no build/test runs, mutation execution, file edits, tool installation, broad suite discovery, or agent delegation. Resolve only the supplied tests, their relevant fixtures/helpers, and the production call chain needed for their claims.

Use the inline test-gap-analysis assessment from Step 2's checkpoint; do not load it a second time. Supply each test's identifier/body, relevant setup/helpers, claimed behavior, assertion semantics, and available source. Its composition dispatch loads the owned read-only reference rather than its standalone baseline/verification workflow. Consume its per-test evidence; do not duplicate its mutation catalog here or invoke an audit/generation agent. Convey mode and inputs as assessment context using the host's supported caller instructions. If the loader accepts only a skill name, load test-gap-analysis by name only; do not invent tool arguments or a mode-specific skill name.

If the skill/reference or production context is unavailable, record Pseudo-mutation: N/A / unverified — <reason> and continue normal body-level grading. This is not a grade deduction or, by itself, an Uncertain result. Do not search installation directories or substitute a mutation runner.

Assess only what each test claims: do not borrow another test's assertions, or demand unrelated branches, outputs, or scenarios. An observable survivor can support an existing Assertion strength category when it proves that the test does not verify its claimed outcome; do not introduce mutation points, weights, ceilings, or an automatic survivor penalty. Apply the existing rubric normally, including weaknesses it classifies in both Assertion and Anti-pattern dimensions; do not add another deduction for the same mutation evidence.

Start every test at grade A (score band 90–100), then apply deductions strictly for observable issues in the captured body. Do not deduct for hypothetical concerns (e.g., "could have more negative assertions") unless the production code clearly demands them and the production code is available.

When production code is unavailable, grade observable issues in the test body normally, but do not infer missing behaviors or deduct for them. State Production-dependent behavior coverage: Unverified once in the summary so the reader can distinguish test-body findings from claims that require source code.

Three sub-dimensions

Compute three sub-grades (each A–F) that together drive the overall grade.

A. Assertion strength

Read the loaded language extension's assertion API list and classify every assertion in the test body. Score from highest to lowest:

Exception and error-path tests (Assert.ThrowsException<T>, constrained pytest.raises, expect(fn).toThrow, assertThrows, #[should_panic], Should -Throw, EXPECT_THROW, or Go code that verifies an expected non-nil error) are complete on their own. Give Assertion strength A when the test checks the exact promised error condition for its stated scope. Do not deduct for having only that assertion, and do not require an error-message assertion unless the message is part of the documented contract. A Go happy-path test that only checks err == nil while discarding a meaningful returned value is still C because it does not verify the successful result.

B. Structure & focus

C. Anti-pattern hygiene

Scan against the catalog below. The Anti-pattern sub-grade is computed in two passes and combined deterministically:

1. Hard ceiling pass. Every Critical or High finding sets a maximum sub-grade (F, D, or C as labeled). Take the worst ceiling across all matched Critical/High findings — these do not accumulate (a single F finding caps the sub-grade at F regardless of how many other Critical/High findings are present).
2. Medium-deduction pass. Start from A, then for each Medium finding deduct one sub-grade level (A→B, B→C, C→D, D→F). These do accumulate across findings.

The final Anti-pattern sub-grade is the worse of the two passes (i.e., min(hard_ceiling, A − medium_count)). Low findings never affect the grade — mention them in the note only.

Examples (Critical/High and Medium counts → Anti-pattern sub-grade):

• Zero Critical/High, 1 Medium → B (A − 1)
• Zero Critical/High, 3 Medium → D (A − 3)
• One C-ceiling (e.g., over-mocking), 0 Medium → C
• One C-ceiling, 2 Medium → C (min(C, A − 2 = C) = C; a third Medium tips to D)
• One F-finding (e.g., swallowed exception) plus any number of Medium → F

Critical (drop straight to F or D)

• No assertions at all → F (also drives Assertion sub-grade to F)
• Swallowed exceptions: try { … } catch { } (.NET), bare except: pass (Python), try { … } catch (e) {} (JS/TS/Java), defer recover() without re-panic (Go), rescue StandardError with no assertion (Ruby), empty catch (Kotlin/Swift) → F
• Assert-in-catch pattern (Assert.Fail(ex.Message) instead of Assert.ThrowsException) → D
• Always-true literal assertions (Assert.IsTrue(true), assert True, expect(true).toBe(true)) → F (verifies nothing; also drives Assertion sub-grade to F)
• Self-referential / tautological assertions on bound values (Assert.AreEqual(x, x), assert dto.name == dto.name) → D
• Commented-out assertions → D

High (drop one or two sub-grades)

• Wall-clock sleep used for synchronization: Thread.Sleep, Task.Delay, time.sleep, setTimeout-based wait, Thread.sleep, time.Sleep, sleep, std::thread::sleep, Start-Sleep, std::this_thread::sleep_for (in a unit test) → D
• Unseeded randomness, wall-clock reads without abstraction (DateTime.Now, datetime.now(), Date.now(), System.currentTimeMillis(), time.Now(), Time.now, Instant::now(), Get-Date, system_clock::now) → D
• Hard-coded environment-dependent paths (C:\…, /tmp/…, network hosts) → D
• Ordering dependency on mutable static / package globals → D
• Broad exception assertion (Assert.ThrowsException<Exception>, pytest.raises(Exception), expect(fn).toThrow(Error) without matcher, #[should_panic] without expected = "…", Should -Throw without -ExpectedMessage, EXPECT_ANY_THROW) → C
• Over-mocking: more mock setup lines than test logic, or verifying exact call sequences instead of outcomes → C
• Implementation coupling: reflection on private members, casting to internal types to access state → C

Medium (drop one sub-grade)

• Poor name: Test1, TestMethod, test, single-word name that says nothing about scenario or expected outcome (judge against the language extension's convention) → drop one sub-grade
• Magic values: unexplained 42, "foo", 0x1234 in arrange/assert without naming or comment → drop one sub-grade
• Giant test (>30 lines covering a single behavior) → drop one sub-grade
• Assertion messages that just repeat the assertion text → drop one sub-grade
• Missing AAA / GWT separation when the test is non-trivial → drop one sub-grade

Low (note only, no deduction)

• Unused setup/teardown hooks; print debugging left in (Console.WriteLine, print, console.log, System.out.println, fmt.Println, puts, dbg!, Write-Host, std::cout); inconsistent naming versus siblings; leftover TODO comments. Mention in the note column but do not deduct.