Figure Remap

figure-remap — deterministic document-grounding system

This skill calls scripts in your clone of the textbook-to-note repo. At install time, replace {REPO} below with the absolute path of the clone.

Architecture (read this before implementation details)

figure-remap is a multi-backend deterministic document-grounding system, not a "crop utility." The system has six explicit layers:

┌─ Contract ──────────────────────────────────────────┐
│  figure_remap.extract() → {status, match_quality,   │
│  hard_fail, file, fig_id, reason}                   │
├─ Policy ─────────────────────────────────────────────┤
│  strict = deterministic + ambiguity-intolerant      │
│  L1 / L2 / L3 hard-fail hierarchy                   │
├─ Junk pre-gate ─────────────────────────────────────┤
│  pregate.verdict() on a QC-passed crop:             │
│  chapter-banner geometry / blank → kill, no retry   │
├─ Backend selection ─────────────────────────────────┤
│  Capability-based: _select_backend()                │
│    "geometric"  | "caption_anchor" | [future]       │
├─ Backends ──────────────────────────────────────────┤
│  geometric_match_bbox()  — born-digital rasters     │
│  caption_anchor_bbox()   — scanned / hybrid scans   │
│  [future] layout-aware / table-aware                │
├─ Analysis & cache ──────────────────────────────────┤
│  PageDerived (per-page, sha1-keyed, policy-versioned)│
│  _render_cache/<sha1>_pg<N>_<dpi>dpi.png            │
├─ Debug artifacts ───────────────────────────────────┤
│  <out>.fail.json on every scanned-mode hard_fail    │
└─────────────────────────────────────────────────────┘

Future backends plug in at the Backend layer with their own capability check and a function returning bbox + ambiguity flags — nothing above changes.

==Architectural freeze rule==: do not add per-book heuristics or if scanned/elif born-digital branches into the gate function. New behavior goes into a new backend plus a new capability check in _select_backend.

Backends — when each is used

_select_backend() routes pages by structural capability, not by document type:

The hybrid-scan category is the one that surprises naive backend selection: the PDF library reports multiple "assignable" rasters (the embedded figure photos), but the surrounding OCR-overlay text contaminates the geometric backend's text-bleed QC check. Detecting the page-background raster and routing to caption-anchor is the fix.

Scanned/hybrid backend — caption_anchor

Module: figure_scanned.py. Algorithm:

1. Render the page at 200 dpi to a cache file (keyed by PDF hash + page + dpi).
2. Walk the page's text-with-position data for Fig X-Y caption text bounding boxes (with an inline-reference filter — must start at a line beginning, not mid-sentence after a word).
3. Apply Class A OCR normalization (silent, 1-to-1): common OCR confusions (l/I → 1, S → 5, O → 0) only inside the numeric portion of the fig_id.
4. Build a caption-match object per caption with an alias set.
5. Look up the target fig_id across all caption alias sets:
   • 0 matches → L1_no_caption_match hard_fail
   • ≥2 matches → L1_target_matches_multiple_captions hard_fail
6. Infer page-global caption direction (caption-below is the layout default).
7. Compute the figure region: the area opposite the caption, bounded by the previous/next caption in the same column.
8. Crop the rendered page; run an ink-density trim to tighten the bbox.
9. Scanned QC: whitespace-fill check only (text-bleed is meaningless on scanned PDFs — the OCR overlay covers the whole page).
10. On hard_fail: write a sibling .fail.json debug artifact with policy version, captions, columns, backend reasoning.

Known Phase-1 limitations (deferred):

• Class B OCR ambiguity (e.g. 4-28 vs 4-2B) currently fails L1_no_caption_match for OCR-mangled targets.
• L2 ownership-overlap guard, L3 direction-quality guard not yet built.
• Direction inference is page-global, not per-column.

fail.json schema

Every scanned-mode hard_fail writes a sibling <out>.fail.json debug artifact, with a versioned schema. Stable fields:

Downstream consumers must check schema_version and refuse to parse unrecognized majors.

Performance helper — extract-page for scanned books

When a workflow needs all figures on one scanned page, use the batch CLI subcommand instead of N independent extract calls:

python {REPO}/figures/figure_remap.py extract-page \
  --book "BookID_PartI" \
  --pdf "<path>.pdf" --page N --out-dir "attachments/topic/" \
  --fig-ids "4-1A,4-1B,4-1C,4-1D" \
  --name-template "Fig_{fig_id}_BookShort.png"

==This is NOT a contract addition== — extract-page is a render-cache-aware loop that calls extract() once per fig_id. Each emitted JSON line is a full single-figure contract dict; workflows can substitute N independent extract calls without behavior change — only speed differs.

Entrypoint contract (call THIS, not the gate)

There is ONE sanctioned entrypoint for every figure request — figure_remap.py extract. Workflows MUST go through it and MUST NOT call the internal gate script directly. The entrypoint is the abstraction boundary: it returns a stable minimal contract and hides the implementation (matching method, strict mode, fallback policy, QC ladder), so logging stays uniform and policy is enforced in one place.

python {REPO}/figures/figure_remap.py extract \
  --book "{BookID}" \
  --fig-id "X.Y" \
  --caption "{caption text from textbook md}" \
  --out "attachments/Fig_X-Y_{BookShort}_{topic}.png" \
  --pdf "{path/to/textbook.pdf}" \
  --page N \
  [--source {auto|pdf|existing}]   # default auto (see "Source modes" below)
  [--existing "{book_dir}/figures/Fig_X-Y.{png,jpeg}"] \
  [--no-strict]      # opt OUT of deterministic mode into the fallback ladder (discouraged)

Source modes

--source selects where the figure comes from. Default is auto — callers don't need to think about a pre-extracted fast path; the entrypoint resolves it from the book convention <OUTPUT_DIR>/<book>/figures/Fig_<id>.* automatically.

Contract returned (and ONLY this — never the gate's internal shape):

{ "status": "pass|fail|escalate", "match_quality": "exact|uncertain|failed",
  "hard_fail": true|false, "file": "<path|null>", "fig_id": "<normalized>", "reason": "<text>",
  "qc_degraded": true|false, "qc_skipped": ["<check name>", "..."] }

Those eight keys, no more and no fewer — _validate() in figures/figure_remap.py raises on any extra or missing key.

• qc_degraded / qc_skipped: one of the two BLOCKING checks could not run (e.g. the source page render was unavailable), so a pass here is weaker than a fully-gated one. Treat a degraded pass as "embed, but eyeball it". A skipped advisory check never sets this — it was never gating anything.
• match_quality: exact = deterministic geometric match · uncertain = a fallback crop (only with --no-strict) · failed = no crop. ==Branch on this, never on the engine method.== The precise method is logged only in a local QC log, deliberately kept out of the contract.
• status:pass (exit 0) → file is a QC-passed crop; embed it. ==Embed the file string verbatim — never the --out you asked for, never a name rebuilt from the Fig_{id}_{Book} convention.== file is the authoritative path and it legitimately differs from --out: the auto fast path returns the book's pre-extracted image, so a template saying .png can come back .jpeg. A rebuilt name is derived from the real one, so it reads correct in review and only a machine catches it. Prove it:

  python {REPO}/figures/figure_embed_lint.py --notes-dir {NOTES_DIR} "<note.md>"

  exit 0 clean · 1 = MISSING (a guessed filename, or an embed written for an extract that actually failed) or CASE MISMATCH (opens on Windows/macOS, breaks on git and Linux) · 3 = notes dir not found = unverifiable, not a pass.
• status:fail (exit 1) → deterministic miss / hard_fail. ==NOT a wrong figure — a correct refusal.== Fix --page, escalate to vision, or leave a <!-- TODO -->.
• status:fail with reason starting pregate= (exit 1, hard_fail:false) → different animal: extraction and QC both succeeded, then the junk pre-gate (figures/pregate.py) judged the crop to be a chapter-title banner or a blank. ==Skip this fig_id entirely — do not retry, do not fix --page, do not leave a retry TODO.== Retrying only re-crops the same banner. Applies to born-digital geometric crops; the scanned caption_anchor path and the existing fast path return no bbox, so the pre-gate abstains there.
• status:escalate (exit 2) → non-strict only; read the page render, then re-call with --bbox.

Default is strict (deterministic). --no-strict re-enables the legacy-compatibility fallback ladder, which can yield a plausible-but-wrong crop — avoid for note-writing.

⚠️ A pass guarantees the right raster for the fig_id, not that the figure depicts what you assume. Still read the real caption text before trusting its content.

Policy version

figure-remap policy (current)

• Deterministic mode (geometric match) is the DEFAULT and MUST NOT be bypassed without an explicit --no-strict.
• The fallback ladder (raw-xref pick / local-vision guidance / size-pick) is legacy compatibility only — never the default path.
• Workflows call figure_remap.py extract ONLY; the internal gate script is private.
• The contract exposes match_quality, not the engine method.
• ==Future changes MUST NOT re-introduce silent fallback (default→ heuristic)== without bumping the policy version. A bug-fix patch must not quietly flip the default. Enforcement: the gate's CLI is guarded, and a regression test suite ({REPO}/figures/test_contract.py) runs after any gate edit.
• extract auto-sweeps ±1 neighbor pages on strict hard_fail before returning fail. Still deterministic (geometric match only, no vision model); just widens the search window by 2 pages. See "Page resilience" below.
• Multi-panel caption auto-relax: when a caption contains ≥2 distinct panel references ((A)(B)… or panel X), strict mode is overridden to relaxed up front — see "Multi-panel figures" below. This is policy-level routing (a capability mismatch, not a silent fallback): strict geometric match's "caption owns exactly one raster" assumption is invalid for composite layouts, so retrying strict can only waste attempts.

Multi-panel figures — capability mismatch

Composite figures with sub-panels (multi-location diagrams, staged classification figures, surgical step sequences) break a core assumption of strict mode and require ==different handling at three levels==: routing, semantic completeness, and uncertainty reporting.

1. Routing (automated)

Strict geometric matching assumes ==one caption owns one raster==. Multi-panel captions reference sub-regions ((A)/(B)/(C)/(D)/(E)) of a composite layout — those panels are not independently owned rasters. This is not an implementation bug; it's a representation mismatch.

Detection triggers on ≥2 distinct (letter) or panel <letter> references (a single (A) is allowed through — common in non-panel prose). When triggered, extract overrides strict=False and records policy=multipanel_caption_relaxed in the contract's reason field.

2. Vector overlay loss (caller awareness)

==Multi-panel figures extracted via raster/xref cropping often lose vector overlays== — panel letter labels, arrows, dotted guides, annotation lines. The crop is technically correct (right page region, right raster) but ==semantically incomplete==: it can be hard to map a sub-region back to its caption text.

Caller doctrine: when embedding a multi-panel figure in a note, write the surrounding description as self-contained — map each panel letter to its content in the prose, so readers can interpret the figure even when overlay labels were lost in extraction.

3. Uncertainty taxonomy (observation)

match_quality: uncertain today collapses three distinct failure modes: identity uncertainty (vision-model fallback selected the raster, not deterministically verified), semantic-completeness uncertainty (overlay labels lost), and geometry uncertainty (crop bbox loose). Caller doctrine: when match_quality:uncertain is returned, prose-annotate the embed (e.g. "vision-extracted, recommend visual confirmation") rather than relying on machine-readable fields.

Implementation: internal gate script (do not call from workflows)

figure_remap.extract() delegates to the internal gate implementation, which handles the ladder + QC in one call (first success wins): 0. geometric_match (primary, deterministic, no vision model) — direction-agnostic caption↔image matching: auto-detects whether captions sit above/below the figure, assigns each embedded raster to its nearest caption, crops the raster(s) owned by the target fig_id (union for multi-panel).

1. Existing file — reuse a prior crop if it passes QC (if --existing)
2. Raw page candidates — largest embedded rasters, decoration-filtered
3. Local-vision-guided retry (×2) — a small local vision model suggests a bbox (weak; last resort before escalation)
4. Escalate — exit code 2 → read the page render, estimate bbox, re-call the gate with --bbox

QC on each candidate: two checks, both pure computation, both block (whitespace fill ≥80%, text-bleed <50 prose chars). No model participates in QC — a model-backed check gave the same crop different verdicts on repeat runs (issue #16), so nothing a vision model says can gate, and the QC chain no longer calls one at all. The only vision call left in the gate is the bbox suggestion in rung 3, which the strict default never reaches; its output is a proposal that must still pass the computed checks, and it is pinned to greedy decoding (temperature 0, top_k 1, fixed seed) so it does not drift between runs. Whether a QC-passed crop is worth embedding is judged downstream by the workflow's frontier-model classification step.

Exit codes: 0 ok (saved to --out) · 2 escalate to frontier vision · other → log + skip.