Lt2md

LT2MD — Long Transcribe to Markdown

LT2MD turns observable PDF content into Markdown that an agent or a person can audit later. It is designed for born-digital, scanned, and mixed PDFs. The goal is not merely to obtain text: preserve reading order, formulas, figure meaning, scope boundaries, and a path back to the source page.

The PDF remains the only authority for content. OCR, extracted text, model guesses, and formatting preferences are candidates or transformations, never evidence that can overrule the rendered page.

Read before doing the task

1. Read workflow.md for roles, batches, two-pass visual reading, and write permissions.
2. Before rendering or reusing pages, read page-cache.md and initialize/recover a local job with scripts/manage_job.py.
3. Before creating or changing a candidate Markdown file, read markdown-contract.md; for long documents also read job-state.md and checkpoint-review.md.
4. Before a format review, read format-review.md and treat 示范文档.md as a read-only format fixture.
5. Use scripts/validate_markdown.py, scripts/audit_markdown.py, and scripts/manage_job.py verify as separate final gates. Do not place OCR, model calls, or PDF interpretation inside the static tools.

Non-negotiable principles

• Separate evidence, semantic target, and allowed transformation. Evidence is the rendered page, PDF page number, readable text layer, and Markdown markers. The semantic target is the author's text, mathematics, figure relationships, and reading order. Allowed transformations include merging print line breaks, removing page furniture, and applying the Markdown contract.
• Lock the scope before writing. If no range is specified, process the whole PDF. If a range is specified, do not silently expand it. A range that ends mid-page includes only the requested semantic blocks.
• Reuse page evidence by byte identity. Render through the content-addressed job cache. The same PDF bytes and render configuration must reuse verified page PNGs across chapters, restarts and renamed files; only missing or corrupt pages may be rerendered.
• Use the rendered page as the tie-breaker. Text extraction and OCR are useful candidates. They do not settle reading order, formulas, captions, diagrams, or ambiguous glyphs without visual confirmation.
• Describe every information-bearing figure. Keep the original caption when readable, then place an adjacent description explicitly marked as a LT2MD/transcriber supplement and not original text. Include objects, labels, directions, arrows, sequence, spatial relationships, subfigures, and relationships directly expressed by the figure without inventing outside conclusions.
• Keep provenance local. Put one block-level SOURCE HTML comment on its own line before every complete paragraph, display equation, figure block, table or example block. Do not insert an anchor inside a word, sentence, inline formula, display-math block, table row, caption or image description. A cross-page block uses one physical-page range before the merged block.
• Keep content and format review separate. Content corrections require evidence from the source PDF. A format reviewer may only report or apply style-only changes against the immutable fixture. Only the coordinator writes the final Markdown.
• Mark uncertainty instead of guessing. When a glyph, page boundary, or reading order cannot be uniquely resolved, give the best source-grounded transcription and add a 转录注 with the exact page and ambiguity. Never silently normalize an uncertain value into a familiar one.

Operating procedure

1. Preflight. Initialize or recover a job. Record the PDF SHA-256, physical page count, requested range, book-page mapping if readable, text-layer availability, render configuration, columns, formula/figure density, output path and task-requirement hash. Render the original pages before trusting OCR. If printed page numbers become visibly clear only after initialization and have a verified linear relation, record it before the first inventory with manage_job.py set-book-page-offset <job> --offset <N>; otherwise retain unmapped rather than guessing. Once recorded, that mapping is source evidence: the batch scaffold's source_print_pages and every SOURCE BOOK_PAGE must follow it, and manager review/checkpoint/finalization rejects contradictions.
2. Batch. Process continuous page ranges adaptively: 1–2 dense/low-quality pages, 2–4 ordinary pages, and at most 6 clear single-column pages. If the user did not choose groups, run manage_job.py batch-plan <job> --json after initialization, then visually lower any recommendation that contains formulas, tables, multi-column order, dense figures, poor legibility, or a cross-page semantic block. The raster-only plan is a conservative starting point, not visual proof. Prefer complete paragraphs, sections, or examples as cut points; keep a sentence crossing a page boundary with one transcriber.
3. Inventory and transcribe. Before trusting any existing Markdown candidate, visually inventory each source page's headings, prose, displayed equations, figures/captions, tables, footnotes, examples/exercises and cross-page continuations. For the current 1–6-page batch, create that source-only record first with manage_job.py source-inventory-template, fill only source objects and evidence, then freeze it with manage_job.py seal-source-inventory. Only after that seal may the transcriber use manage_job.py batch-template --author-id <transcriber> to create a fresh, non-overwriting batch-scoped candidate. This order is a hard gate: candidate block IDs, review decisions and candidate text must not be retrofitted into the source inventory. Never copy an unreviewed full-document V1 draft into the batch candidate and mistake a whole-document audit failure for a batch transcription attempt. Separate body text, equations, figures, captions, examples, headers, footers, and scan noise. Preserve literal Markdown backslashes while writing formulas: an escape-interpreting string layer must not turn a formula command into TAB, FF, or another C0 control byte. Merge only print line breaks and cross-page continuation; do not insert a page boundary inside a word, sentence, or LaTeX expression. An existing Markdown draft is an untrusted candidate, not evidence: visually re-check every retained block. If an inventory item has no source-grounded candidate block, leave the batch blocked; do not omit it merely because the candidate lacks an anchor. If a block is left unchanged, preserve page-specific review evidence; if the page cannot be read, stop there rather than calling the unchanged draft complete.
4. Coordinate. Merge candidate blocks in source-page order, attach page anchors, preserve equation tags and figure/example structure, and keep the locked range visible.
5. Second visual read. After the candidate passes its static contract and before generating a review template, record a handoff of its exact bytes with manage_job.py reviewer-handoff. The manager, not reviewer-supplied JSON, owns the reviewer actor ID, local security-principal record, candidate digest, and sealed-inventory binding. The default policy is an auditable process handoff: it does not prove subjective independence merely because labels differ. An optional init --review-identity-policy os-security-principal-v1 also requires the reviewer process to use a different local OS security principal from the candidate and source-inventory authoring processes; it still cannot prove distinct people or model contexts. The handoff reviewer re-reads the rendered source and completes mappings against the already sealed source-only inventory. The reviewer may add candidate mappings, dispositions and risk closures, but may not rewrite sealed source facts. The coordinator changes content only after confirming the source. An omitted footnote, caption, heading, or cross-page continuation remains blocking even when static Markdown checks pass.
6. Risk-driven third read. Run the audit and re-check only real differences, low-resolution areas, dense formulas, multi-panel figures, cross-page joins, scope boundaries and risk hits. Use only the cached target/adjacent pages and targeted crops.
7. Checkpoint and recover context. Freeze every complete 1–6 page batch with a source-bound checkpoint-review JSON manifest through manage_job.py checkpoint before starting later pages. Use manage_job.py review-template only after the sealed-inventory-backed candidate passes the static contract and its exact-byte reviewer handoff is recorded; it produces a blocked identity/hash scaffold and does not replace source review. The manager rejects a missing handoff, a stale candidate digest, a forged reviewer label/principal, an indented-code pseudo-anchor, a review block spanning multiple SOURCE blocks, or a structural modification hidden by whitespace normalization. A failed static check, audit, source-inventory mapping, or independent review is a stop condition: repair the same batch or leave it explicitly incomplete; never treat a failure report as permission to continue. If a source object visibly continues to the next physical page before any independent review, do not accept the short batch or anchor a fragment. Use manage_job.py extend-unclosed-source-inventory only to preserve its sealed source facts and exact unclosed candidate while expanding the same-start range to at most six pages; then re-inventory every page, create a fresh candidate, and complete the normal independent review. This extension is blocked evidence, never acceptance, and cannot change a checkpointed range. When a reviewer supplies source-grounded omissions, misreads, ordering defects, or wrong-page anchors, return only that batch and the exact evidence to the transcriber, then obtain a new independent reread—never relabel the old review as accepted. If that review proves the source-only inventory facts themselves are incomplete or wrong, do not mutate the old seal: use manage_job.py source-inventory-revision-template with that independent blocked review, reread and seal the new source-only inventory, then create a fresh replacement candidate for the same range. The manager freezes a SHA-named copy of the blocked candidate and review; the new candidate receipt must bind the new active seal, and verification checks both the forward and backward revision chain. It rejects self-review, stale candidate replay, altered lineage, and unchanged source facts; a blocked review can never become acceptance. For jobs created before frozen-candidate evidence existed, use the strict manage_job.py backfill-revision-evidence <job> --pages <range> migration only when the preserved bytes, hashes, receipt, old seal and blocked review agree exactly. Every 16 accepted physical pages or 4 accepted batches, whichever comes first, actually reread task brief, render manifest, progress, frozen evidence and risk queue, then record the receipt with manage_job.py reread. This longer cadence supplements, rather than replaces, the per-batch evidence checkpoint; a cache-only or partial job must not pass manage_job.py verify.
8. Format review. Assign one format reviewer per candidate. The reviewer compares only Markdown form with the read-only fixture and returns FORMAT_OK or strict FORMAT_CHANGE records. The reviewer must not read the source PDF or change content.
9. Static validation. Save a pre-format snapshot, run the validator, audit and job verifier, and when style-only repairs were applied run the validator again with --before <snapshot>. All content projections and source-anchor values must remain unchanged.
10. Deliver. Only the coordinator atomically replaces the final Markdown after all gates pass. First make a blocked full-range final-review-template, obtain an independent source-page acceptance, then use manage_job.py finalize; verify with manage_job.py verify --require-finalization. If interruption leaves a prepared promotion, do not start another finalization: run manage_job.py recover-finalization <job> and re-verify. Report the final Markdown path, converted range, cache/job identity, validation result and any 转录注. Do not deliver internal candidate drafts unless the user asks for them.