Perch

perch

perch asks a model about the code in a repository. It reports what it believes, as a probability on every finding.

Scan what changed

$ perch scan --since origin/main
checkout.py
  ID        Line  Severity  Type    Confidence  Problem      Method
  bdc67421    14  P1 (0.8)  defect         81%  wrong_order  place_order

✖ 1 problem in 1 file, all failing
perch at commit 5e9d910: 3 methods, read 3
3 requests  10k tokens in / 2k out  $0.0004

--since <ref> covers what moved since that ref. That is what you want on a branch and in CI. --paths a,b covers named files or directories. Add --json for the full detail: perch scan --since origin/main --json.

A whole repository is hundreds of model requests. A branch is a handful. Never scan everything to check one change.

A method whose code and neighbours have not moved is skipped next run. Scanning again after a fix costs almost nothing.

3 is a result rather than an error, so report what it found. Only 1 and 2 are failures.

Read the JSON

Every command takes --json. The table is rounded off for a terminal, and the JSON carries the detail.

$ perch issues
ID        Method             Location           Type      Kind                    Severity
05a5b5bd  scanRepository     src/scan.js:129    refactor  too_big 98%             -
b2d0885f  formatFinding      src/report.js:376  refactor  too_big 98%             -
990c94ac  scan               src/cli.js:350     refactor  too_big 92%             -

perch issues <id> --json gives one finding in full. Every question's answer carries the distribution behind it:

"has_bug": 0.46,
"kind": { "choice": "missing_null_handling", "probability": 0.84,
          "probabilities": { "missing_null_handling": 0.84, "wrong_return": 0.08, "boundary": 0.04 } },
"severity": { "level": "P2", "score": 1.4, "confidence": 0.38,
              "probabilities": { "0": 0.09, "1": 0.43, "2": 0.47, "3": 0.01 } }

Reading a finding

A finding is an advisory drawn from a probability and a rule. It does not locate a defect for you.

The number is belief. How bad the problem would be is a separate field. 72% means the model would say yes about seven times in ten.

Probabilities spread across several kinds mean the model is sure something is wrong and cannot say what. Say 0.4 on one kind and 0.3 on another. Read the method. Leave the label alone.

The line it points at carries its own confidence. A low one puts the problem somewhere in the method.

Expect the real problem to sit adjacent to what was reported. A missing_null_handling at 70% is often an unchecked error a few lines off.

Answers below a question's min stay out of the report and stay in the JSON. A 56% has_bug is worth a look while the report is silent about it.

Read the code before you change it. Close a finding when the code is right. Never rewrite working code to satisfy a probability.

Check one method

After a fix, ask about the thing you changed.

$ perch check src/store.js::openStore.scanDir
src/store.js:74  openStore.scanDir
5 checks, 0 broken.

  Also raised: does_not_do_what_it_claims 80%
2 requests  5k tokens in / 767 out  $0.0002

perch check src/store.js::openStore.scanDir --json gives the same reading in full, and an issue id works in place of a target.

This reads the file off disk, so it works on uncommitted code. It records nothing, so it will not move the numbers on the issue you are fixing. It exits 3 while something is still wrong.

Close a finding you have judged

$ perch close 05a5b5bd --reason "the walk is one job read top to bottom"
05a5b5bd  scanRepository  src/scan.js:129  closed  too_big

A close covers the kinds the issue was listing. A different problem found on that method later is still reported. --kind too_big closes one kind and leaves the rest open.

Always give a reason. That is what the next person reads instead of reopening it.

Check the tests a change needs

perch coverage --since main runs predictive mutation testing over the branch. It makes every mutant of every method a test reaches, from a removed call to a flipped comparison, and asks which tests would fail against each mutant. It runs no tests and reads nothing a test run wrote.

$ perch coverage --since main
src/cart.ts
  ID        Line  Problem   Confidence  Test or method  Note
  3945b1f7    12  survived         60%  applyDiscount   With `>` instead of `>=`, none of the 5 tests reaching it…
  d6f68653    12  survived         58%  applyDiscount   With `1` instead of `0`, none of the 5 tests reaching it …

test/cart.test.ts
  ID        Line  Problem    Confidence  Test or method              Note
  427fce7a    22  redundant         96%  applyDiscount > takes 75 …  Kills the same mutants as applyDiscount > ta…
shop at commit 861c607: 4 methods, 12 tests, 3 problems in changed code, 4 elsewhere
Report: .perch/coverage/index.html
14 requests  12k tokens in / 666 out  $0.0036

A survived mutant is a line the tests run but never check. --json gives each one in full: the finding names its mutant, and the method's mutants entry with that id has the line as written (original), as mutated (mutated), and the tests that were asked (asked). Write a test in one of those test files with an input for which the two lines give different results, and assert on the result. For if (percent >= 100) against if (percent > 100), that input is percent of exactly 100.

A redundant test kills exactly the mutants an earlier test kills: compare the two and delete one. checks_nothing kills no mutant in the code it reaches: make it assert on what that code returns. infra calls a live service: mock it.

Run perch coverage --since main again afterwards; the mutant should be gone from the list. It exits 3 while a problem remains in changed code. Close a problem with perch close <id> --reason "..." when the test is right as it is.