Audit Sft Data Quality

Audit SFT Data Quality

Core rule

Judge every row against the target task. A polished answer is not high-quality supervision if it teaches the wrong behavior, violates the task contract, leaks the evaluator, or cannot be verified.

Apply hard correctness and integrity gates before diversity scores, confidence scores, or aesthetic judgments.

For base-eval diagnosis, capability planning, synthesis, augmentation, and iterative dataset design, read iterative-sft-data-design.md.

1. Define the behavior contract

Write the contract before reading candidate answers:

Do not infer train readiness while any contract-critical field is unknown. Record assumptions explicitly when the source does not define them.

2. Freeze the source inventory

Preserve immutable evidence before transforming data:

• source path, repository, revision, archive member, or URL;
• file and member SHA-256;
• row counts and unique task counts;
• schema version and split;
• synthetic, human, model-generated, repaired, or imported provenance;
• license, privacy, consent, and secret-handling constraints;
• parent row or revision lineage.

Never overwrite raw inputs. Put normalized, selected, repaired, and rejected rows in separately identified artifacts.

3. Validate structure and conversation semantics

Check every row, not a sample:

• parseability and required keys;
• stable task identity and label consistency;
• legal role order and nonempty assistant target;
• output-format compliance;
• tool-call and tool-result pairing;
• referenced files, attachments, schemas, and environments;
• absence of accidental test, reference-answer, private-state, or system-prompt content;
• tokenizer-measured length under the actual model revision;
• loss masking and target boundaries when the training loader uses them.

Reject malformed rows rather than silently coercing them unless the repair is deterministic, recorded, and reverified.

4. Categorize and tag every row

Use tags that support balancing, diagnostics, and regression analysis. Include:

• task family and domain;
• target capability;
• output mode and interaction mode;
• algorithm, tool, or reasoning pattern;
• stateful versus stateless behavior;
• edge-case and failure-mode tags;
• difficulty and resource profile;
• source, split, revision, and synthetic status;
• verification status and oracle type;
• safety, privacy, or contamination risk.

Prefer explicit tags such as mutation-atomicity, boundary-conditions, tool-repair, deterministic-ordering, or calibrated-abstention over vague labels such as hard or quality.

5. Detect duplicates and lineage

Compare within the candidate set and against all existing sets at multiple levels:

1. exact row hash;
2. exact task ID or label;
3. normalized prompt and answer;
4. prompt-only and answer-only matches;
5. revision suffixes and explicit parent metadata;
6. source slug and source hash;
7. template, paraphrase, or semantic overlap;
8. equivalent tests, APIs, or solution contracts under different names.

Classify matches instead of calling all of them duplicates:

Do not train an ancestor and its correction as independent examples unless the training format explicitly teaches critique and correction.

6. Audit contamination and split integrity

Check more than exact task IDs:

• benchmark names and public exercise IDs;
• prompt, title, filename, symbol, and test similarity;
• answer or reference-code similarity;
• shared templates and generated variants;
• parent-child lineage across splits;
• hidden labels, expected outputs, tests, rubrics, or judge feedback;
• near-duplicate domains that preserve the same executable contract.

Use group-aware splits by underlying task, template, source family, and revision lineage. If a row teaches an evaluation answer directly, remove it from training or move the evaluation surface.

7. Verify correctness with the strongest oracle

Prefer deterministic evidence. Use the strongest applicable method:

Run tests answer-blind where possible. A compile pass proves syntax, not semantic correctness. A reward, confidence score, or judge approval does not override a deterministic counterexample.

Use model judges carefully

• Give the judge the task contract and candidate, not the desired verdict.
• Require a verdict, confidence, evidence, and concrete counterexample.
• Use independent judges or adjudication for uncertain and high-impact rows.
• Separate specification ambiguity from answer failure.
• Send disagreements and low-confidence rows to review.
• Never make judge approval the sole hard gate when executable evidence exists.