Zhihu Instrument Test Governance

Zhihu++ Instrument Test Governance

回归与数据边界

新增回归用例前必须在基线得到可复现失败，再在修复版本用同一用例验证通过。不得为注入时间或伪造后端数据新增一次性生产 helper；线上数据变更必须走真实服务端写入与读回，不能用 seed 冒充。

Treat emulator time as a recurring maintenance cost. Keep it only where a real product contract or historical regression needs Android, Compose, window, lifecycle, accessibility, or device integration to prove the behavior.

Non-negotiable Rules

1. Never delete a test that protects a bug which actually occurred.
2. Every retained @Test must have verified issue and fixing/introducing PR links next to it. Never invent provenance.
3. Keep test cleanup out of feature and bug-fix PRs. Use a dedicated branch, worktree, commit, and PR.
4. Do not add a permanent instrument test merely to confirm a low-risk visual choice once. Build the APK and capture a real before/after screenshot instead. A temporary instrument test may capture a screenshot for review. After retrieving the image, restore the test source to its original state before committing or submitting a PR. Permanent tests may capture pixels in memory only when those pixels support a meaningful visual assertion; do not retain screenshot file output or a screenshot-only assertion.
5. Do not run the complete instrument suite locally unless the user explicitly asks. Prefer compilation, the smallest relevant test, or GitHub CI.
6. Flakiness, runtime, or inconvenience never justify deleting a proven regression test. Repair its synchronization, fixture, or execution boundary.
7. A test must assert user-observable behavior or a proven regression, not copy an implementation constant list into assertions. Delete tests that only restate the current source shape without an independent contract; validate the behavior through real interaction, build, or focused regression evidence instead.
8. Every retained regression test must be red against the pre-fix implementation. Temporarily restore the old behavior (or run the test against the parent revision) and record the failing assertion before accepting the test.
9. Every retained test must document both the related issue and the fixing or introducing PR with full URLs in its provenance KDoc.
10. Every retained test must state the target state, what it verifies, and why the assertion protects that behavior; a bare test name or implementation-detail assertion is insufficient.
11. Before stopping, execute every changed regression test against both the pre-fix implementation (must fail) and the fixed implementation (must pass). Record the exact red/green evidence; compilation alone is not sufficient.
12. Every PR that adds or changes tests must state in its body, per test, whether red-to-green verification was completed, including the commands and terminal results or an explicit blocker.

Read references/provenance-and-decisions.md before changing tests.

Required Provenance Block

Place a KDoc block immediately above the annotations for every permanent test:

/**
 * Regression: https://github.com/zly2006/zhihu-plus-plus/issues/123
 * Fixed by: https://github.com/zly2006/zhihu-plus-plus/pull/456
 */
@Test
fun restoredBehaviorStaysStable() = Unit

For a feature contract rather than a reported regression, use:

/**
 * Contract: https://github.com/zly2006/zhihu-plus-plus/issues/123
 * Introduced by: https://github.com/zly2006/zhihu-plus-plus/pull/456
 */
@Test
fun featureContractStaysStable() = Unit

Both URLs are required. A commit hash, branch name, issue number without a URL, current cleanup PR, or a guessed related report is not historical provenance.

An introducing PR without an issue is not enough. A smoke test has no exemption. If no verified pair exists, record the test as UNVERIFIED in the audit. Do not add a fake block merely to make a checker pass, and do not create a retrospective issue solely to legalize an existing test.

Workflow

1. Establish an isolated cleanup surface

• Fetch the intended base and create a dedicated worktree from it.
• Record the base SHA.
• Inventory staged, unstaged, and untracked files before editing.
• Do not touch a feature worktree or a dirty main checkout.
• Never set or change GRADLE_USER_HOME.
• Never run gradle --stop or ./gradlew --stop.
• If a task-owned Gradle process must be stopped, first prove the exact PID, cwd, and command belong to this worktree; terminate only that process.

2. Build a per-test audit

Inventory every @Test under app/src/androidTest. For each test record:

• file and test name;
• verified issue URL;
• verified fixing or introducing PR URL;
• classification: REGRESSION, CONTRACT, SMOKE, or UNVERIFIED;
• why an Android device is necessary;
• overlapping cheaper coverage;
• decision: keep, migrate, consolidate, or delete;
• evidence for that decision.

Use git history as the starting point, then read the linked issue and PR. Search by the test name, nearby production symbols, commit SHA, and user-visible behavior. A filename containing an issue number is a clue, not proof.

3. Apply the deletion gate

Use this order for every test:

1. If it covers a real historical bug, keep it. Consolidation is allowed only when another retained test proves the same regression at least as strongly and carries the same provenance.
2. If it protects a current product contract, keep it only when the assertion genuinely needs Android/device behavior.
3. If the contract is valuable but device execution is unnecessary, migrate the coverage to the cheapest appropriate JVM/common test before removing the instrument version.
4. If it only captures a one-time visual preference, implementation detail, duplicate fixture plumbing, or behavior already proven more strongly elsewhere, remove it.
5. If provenance or equivalence is uncertain, keep it pending more evidence. Uncertainty is not permission to delete.

SMOKE is an audit classification, not an exemption. A smoke test can remain permanent only when a verified issue and PR document why that device-level signal matters. A pure reducer, mapping, or list transform cannot be retained by renaming it a smoke test.