Aic Collector Op Development

AIC Collector OP Development

Build the smallest Collector change whose benchmark invocations are valid and whose persisted rows satisfy the existing AIC consumer contract.

Required reading

Before editing, read:

1. .claude/rules/collector/layer_permissions.md, .claude/rules/collector/failure_handling.md, and .claude/rules/collector/case_authoring.md — the repository-owned policy. It is authoritative over anything restated in this skill.
2. collector/README.md
3. collector/cases/README.md
4. docs/perf_database/collector-v2-population-design.md, especially Three identities, Population flow, and Safe deduplication rules
5. The relevant collector/<backend>/registry.py, collector module, and model/base case YAML
6. Every Python and Rust loader/query that consumes the op's perf file

If the task proceeds to GPU collection, also use $aic-auto-collect.

Non-negotiable rules

• Treat Collector output as an existing-consumer contract. Do not modify SDK, Rust, EngineSpec, or Dynamo Planner merely to make a generated case useful. Consumer changes require separate explicit scope.
• Do not call a case invalid merely because the synthetic collector cannot run it. If the framework and consumer support the shape, fix or record the Collector gap instead of pruning coverage.
• Do not restore coverage with a different kernel under the same logical label. Persisted quant/backend labels must describe the invocation that actually ran.
• Do not modify historical snapshots to make an alignment test pass. Use read-only comparisons as review evidence.
• Do not add permanent legacy_* runtime concepts for a one-time migration. Put production defaults in normal base-op YAML and keep migration comparisons outside the runtime schema.
• Do not ship a deduplication path unless current repository-owned YAML produces at least one duplicate invocation. Record that stage's before/after counts and prove that the unique invocation and persisted-key sets are unchanged. Remove no-op deduplication instead of manufacturing duplicate-only fixtures for it.
• Treat the public getter and any subprocess or inner re-expansion as one population contract. Full/raw and targeted entry paths must consume the same resolved YAML quantization and SM gates; do not reconstruct the policy with a second hard-coded hardware heuristic.
• A case-population change may touch collector/collect.py only for model/op plan selection required by the resolved case plan. Generic resume, retry, checkpoint, logging, and output-finalization behavior require separate explicit scope.
• Anchor collector behavior to the requested framework version. Prefer one verified __compat__/version route over speculative multi-version branches.

Step 1: Establish the consumer contract

Search from the perf filename and op name through both producers and consumers:

rg -n "<op_name>|<perf_filename>|query_<op>" collector src rust tests
rg -n "PerfFile|PerfDataFilename|log_perf" collector src rust

Record a temporary contract table; do not commit it unless it is useful product documentation:

Answer before coding:

• Which columns form the exact lookup key?
• Which axes interpolate, and which require an exact bucket?
• Which TP, EP, head, window, quant, dtype, phase, and backend values can the consumer request?
• Does a missing key fail, use an empirical fallback, or silently select another path?
• Does the op have a production consumer? If not, keep it registry-only or explicitly experimental instead of adding it to default model plans.

Never assume an adjacent key is a fallback. Verify it in the loader/query code.

Step 2: Define three identities

Keep these separate:

1. Recipe identity: why YAML requested the work.
2. Benchmark invocation identity: every value that can change the executed model, kernel, runtime setup, or quantization.
3. Persisted physical key: the columns used by current consumers.

Deduplicate only when both invocation identity and persisted key are equivalent. A persisted-key collision is a bug unless the invocations are proven identical. If repository-owned cases map distinct invocation identities to one consumer key, fail population with the conflicting owners and key. Do not silently use first-wins or widen the SDK/Rust schema inside a Collector-only change.

Examples:

• Model aliases may collapse for a shape-only synthetic benchmark.
• Checkpoint paths must remain separate while native quantization or module behavior is path-dependent.
• W4A16 and W4A8 are distinct when activation precision changes.
• NVFP4, MXFP4, FP8, and INT4 labels are not interchangeable because geometry happens to match.

Step 3: Design cases without accidental Cartesian products

• Keep independent workload axes such as batch or token count as lists.
• Keep correlated structural axes in one profile. Typical correlated fields are (heads, KV heads, head dimension, window, TP) and (experts, top-k, hidden, intermediate, TP, EP, quant).
• Put shared production defaults in collector/cases/base_ops/<op>.yaml.
• Put model-native topology and artifact policy in collector/cases/models/*_cases.yaml.
• Make targeted structural population exact when a model profile exists; it may still reuse shared workload sweeps. Full/raw collection may union defaults and model profiles, then stably deduplicate.
• Use stable first-wins deduplication on the real invocation/key identity. When equivalent recipe representations collapse, document the canonical representative (for example the smallest TP for a local-head key).
• Add a generic synthetic default only when a consumer or interpolation need is demonstrated.

For an existing op, compare current and candidate physical key sets during the change. This is change-specific evidence, not a permanent compatibility mode. Report kept, added, removed, and deduplicated separately; totals alone cannot reveal coverage loss. Migration baselines, exact V1 totals, and historical snapshots are review evidence, not permanent unit-test contracts unless an unchanged consumer explicitly depends on that exact inventory.

Step 4: Implement one vertical slice

Touch only the layers the op needs:

1. Base/model YAML and model-plan selection
2. Case generator/getter when YAML cannot express the required correlation
3. Backend collector implementation
4. Backend registry and exact framework version route
5. Persisted schema/logging
6. Focused tests for generated cases and consumer-visible keys

Before adding a helper, schema field, or filter—and again before finalizing—find its producer in current repository-owned YAML and its reader on a production population path. If a later design decision removes either side, delete the orphan. Do not retain speculative aliases, phase selectors, compatibility modes, or one-time audit scaffolding.