Corpus Building

Corpus Building

The core problem

A folder of PDFs is not a corpus. A corpus is a collection of addressable evidence units — each one traceable to a specific page of a specific source, with enough context to be used in writing without reopening the original document.

This skill covers the full path from source object to evidence card, including OCR, translation, quality scoring, and the noise problems that appear at scale.

Key concepts

Source card — one record per registered source. Holds metadata, processing status, and links to all derived evidence.

Text slice — one evidence unit extracted from a specific page range of a source. Stores the original text, cleaned OCR text, and working translation as separate layers. Never collapses them.

Image slice — one image evidence unit (a page scan, figure, plan, photograph, table, or diagram). Stores file path, description, and status.

The separation rule: a slice stores what the source says. AI annotations store what AI thinks about the slice. Do not mix them.

Workflow

1. Create the source card first

Before creating any slices, create a source card for the registered source object. Minimum fields: source ID, title, language, file path, topics, status fields for OCR and translation.

2. Run OCR

For each source, generate per-page text output.

Default path: use a local OCR engine (Tesseract or equivalent). For multilingual documents, set the language pack explicitly — running an incorrect language pack produces script-mismatch noise, not just bad quality.

Cloud OCR path (optional): use a cloud OCR API for higher fidelity on difficult scans. Reserve this for pages that local OCR consistently fails on, not as a default for all sources.

Output per page: raw OCR text file. Keep raw output separate from any cleaned version.

3. Slice into evidence cards

For each page (or logical text unit), create a text slice card with:

• Slice ID (SLC-####-T###)
• Source ID and page reference
• Original text (from source)
• OCR-cleaned text (if different)
• Working translation (in your project's working language)
• Status fields: OCR status, translation status, verification status

For each page image, create a baseline image slice card with:

• Slice ID (SLC-####-I###)
• Source ID and page number
• Image file path
• Image type (page scan, figure, plan, photo, table, diagram)
• Description status

On image descriptions: use a multimodal model to inspect the actual image pixels. Do not use OCR output as a substitute for visual inspection. Label OCR-derived descriptions as ocr_assisted_draft — they are lower trust than a genuine visual description.

4. Run quality scoring

After slicing, score each page on:

• OCR quality: are there repeated characters, symbol noise, script artifacts?
• Translation quality: is the working language layer present and readable?

Assign a triage priority:

• P1 — likely broken; inspect immediately before using
• P2 — needs review; often image-heavy or missing translation
• P3 — usable but worth spot-checking
• OK — acceptable for browsing

Calibrate before scaling: after your first new source type, inspect all P1 pages plus a sample of P2 and OK. If the scoring looks wrong, fix the scorer before running 50 more sources.