Dedupe

Dedupe

Produce a proof-rich duplicate report for a bounded collection. Keep discovery, grouping, canonical selection, mutation, and verification separate so unique information cannot disappear silently.

Runtime composition

Apply these owners for every dedupe task. Load their current instructions instead of copying their procedures here.

1. Load always-current-datetime and acquire its clock anchor before a user-facing reply. Query runtime time again only when freshness, age, redirects, file times, or a newest-item tie-breaker affects the decision.
2. Load starting-point. Fix the requested result, bounded inputs, authority, required evidence, unknowns, and forbidden outcomes.
3. Load outcome-bounded-work. Treat adapter steps as candidate routes unless a step protects identity, safety, authorization, reproducibility, or proof.
4. Load logic-audit. Audit identity rules, normalization, similarity thresholds, non-transitive pairs, conflicts, canonical tie-breakers, nulls, missing values, links, and edge cases before mutation and completion.

Stop with contract-unavailable: <skill-name> if a required owner cannot load. Never replace an unavailable owner with remembered text.

Commands

apply names the reviewed workflow phase, not a script command. The bundled script does not expose an executable apply subcommand. Carry out an approved plan only through an available host capability after the exact source, canonical targets, actions, conflict handling, and destination are authorized.

Map a destructive request to inspect first unless a current, reviewed report and exact mutation authority are already present.

Required inputs

Before inspection, establish:

• bounded source items and the dedupe unit
• one adapter: text, file, record, url, list, or skill
• identity class: exact, normalized, or similarity candidate
• normalization and comparison policy
• canonical-selection rule
• conflict and merge policy
• mutation authority and destination
• proof threshold and rollback or reconstruction path

Do not invent a missing identity, normalization, threshold, or tie-breaker. Ask only when the missing fact cannot be retrieved and would change the result.

Identity classes

Keep these classes separate in every report:

1. Exact identity: byte, value, or packet equality under a deterministic representation.
2. Normalized equality: equality only after the disclosed normalization policy.
3. Similarity candidate: a scored pair from an explicit algorithm and threshold. It is not identity and does not reduce the canonical count without review.

Similarity is generally non-transitive. Never turn A≈B and B≈C into A=B=C unless the selected algorithm proves an equivalence relation.

Procedure

1. Enumerate only the requested scope with stable source indices and provenance.
2. Load the matching adapter reference before choosing keys or normalization.
3. Run exact comparison before normalized or similarity comparison.
4. Group equal comparison keys. Preserve unresolved items outside groups.
5. Select one canonical item by the declared rule. Default to first-seen only when source order is meaningful and the user accepts it.
6. Record conflicts and unique fields. Conflicts are not removable duplicates.
7. Produce the report before any write, delete, replace, move, merge, redirect lookup, or persisted change.
8. Build a mutation plan from reviewed groups. Map every affected item to its canonical item and state retained information.
9. Apply only the approved plan. Do not widen scope or recursively clean related collections.
10. Re-enumerate the source, reconcile counts, verify hashes or values, and rerun inspection. The second run must be a no-op for the approved identity class.