Textbook To Md

Textbook-to-Markdown Converter

This skill calls scripts in your clone of the textbook-to-note repo. At install time, replace {REPO} below with the absolute path of the clone.

Purpose

Convert PDF or EPUB textbooks into searchable markdown that the agent can grep/read directly, eliminating the need for PDF-library extraction at every query.

Output lives outside the note vault, at the path configured by OUTPUT_DIR in shared/config.py (default ./output/), and is for the agent's consumption, not for the user's reading.

Figures are on-demand, not pre-extracted. This skill produces markdown text only. Figures are extracted one at a time when a note needs them, via the figure-remap skill's entrypoint (QC-gated). Do not batch-extract a whole book's figures into a figures/ folder — that approach does not scale and is unnecessary since the on-demand path already handles it. A legacy figures/ folder may exist for books converted before this design; new conversions are markdown-only.

When to use

• User explicitly asks to convert a textbook or chapter
• You need to search textbook content and want to avoid per-query PDF re-parsing overhead
• Building up a knowledge base from a personal library of reference books
• Before starting work on a new topic, convert the relevant chapters first

Quick reference

# Single file
python {REPO}/converter/convert.py "path/to/chapter.pdf"

# Single file with custom output and label
python {REPO}/converter/convert.py "path/to/chapter.pdf" "output.md" --book-label "Author Title 2e — Ch32"

# Batch-dir: convert ALL PDFs in a directory tree (auto chapter split + PDF bookmarks)
python {REPO}/converter/convert.py --batch-dir "path/to/your/textbook/folder"

# Force re-convert (ignore existing md)
python {REPO}/converter/convert.py --batch-dir "path/to/your/textbook/folder" --force

# Force OCR for ALL PDFs in batch-dir (bypass the text-extraction path entirely).
# Use when the text layer "looks" healthy but quality is actually bad (OCR-overlay
# scans, some digitized reprints) — auto-detection won't trigger because the text
# layer passes the shallow check.
python {REPO}/converter/convert.py --batch-dir DIR --force --force-surya

# EPUB → markdown (the 2nd arg is a FOLDER, not a .md file)
python {REPO}/converter/convert.py "path/to/book.epub" "<OUTPUT_DIR>/Author_Title_2e_2022"

Batch mode skips files whose markdown already exists and is newer than the source PDF. Batch-dir mode saves progress to batch_progress.json — if interrupted, re-running resumes where it stopped. --batch-dir also picks up .epub files automatically.

EPUB support

• Uses pandoc (epub → gfm). EPUBs are reflowable, so there are no <!-- page N --> markers; files carry a <!-- SOURCE: epub --> marker instead.
• If pandoc emits proper heading markup, that drives the chapter split. If the EPUB is CSS-styled with no semantic headings (common), headings are rebuilt from the TOC link table + body anchors before splitting.
• Output is a folder of chNN_*.md + full_text.md (same shape as a PDF book) — the single-file 2nd argument is the output directory, not a .md path.
• Requires pandoc on PATH. Figures are not extracted from EPUB (the on-demand figure flow is PDF-only).

Batch-dir features

• Recursively finds all PDFs, auto-outputs to <OUTPUT_DIR>/{PDF_stem}/ (OUTPUT_DIR from shared/config.py; default ./output)
• Produces full_text.md (complete with <!-- page N --> markers) plus a chapter split
• Chapter-splitting priority: PDF bookmarks → pattern detection → force-split (every 30 pages)
• Writes PDF bookmarks to source PDFs if none exist
• Auto-routes OCR-needed PDFs to the local OCR engine (GPU). Two triggers: (1) a page is scan-only (near-zero extractable characters); (2) fitz-silent-failure is detected (see docs/ocr-ladder.md). Falls back to skip-with-explanation if the OCR environment is missing. Output is full_text.md with page markers, and a chapter split is attempted on the OCR'd text too (numbered-heading patterns often survive OCR); it is best-effort and simply yields no chapters when heading detection fails

Output structure

<OUTPUT_DIR>/                       ← default ./output (shared/config.py)
├── Author_Title_Edition_Year/
│   ├── ch01_Chapter_Title.md      ← conversion produces md only
│   ├── ch02_....md
│   ├── full_text.md
│   └── figures/                   ← LEGACY ONLY — not produced by conversion
└── Another_Book/                  ← single-file conversions use the same <book>/full_text.md layout

Figure-registry generation (figure_registry.json) is an optional external hook, not a shipped script — set the FIGURE_REGISTRY_SCRIPT env var to a generator script if you have one; if unset, post_convert.py skips that step.

==Conversion writes .md files only.== A figures/ subdirectory is legacy — present only on books converted before the on-demand switch. New conversions never create one; figures are pulled on demand by figure-remap.

What the converter produces

Each markdown file contains:

• Page markers: <!-- page X --> at every page boundary, to map content back to the source PDF
• Cleaned text: control characters removed, broken/hyphenated words rejoined, grep-searchable
• Tables: extracted as markdown tables (a low content-cell threshold is used to avoid discarding real tables)
• Figure/table reference markers: when the text mentions a figure or table (Fig. 32.1, Table 31.2, etc.), an HTML comment <!-- REF: Fig. 32.1 → see PDF page X --> is inserted so the agent knows where to look in the original PDF

Searching converted textbooks

Two search methods:

1. Keyword search (grep) — exact match, 0 tokens

grep -r "your search term" "$OUTPUT_DIR"      # default ./output

2. Semantic search (optional) — concept match

If you've set up the optional semantic index (LanceDB + a local embedding model — see docs/architecture.md), a textbook_search tool becomes available for concept-level queries that don't depend on exact wording.

Search strategy

• Known keyword → grep first (fastest, exact)
• Concept/topic exploration → semantic search (finds related content even with different phrasing)
• Both → grep for precision, semantic search for coverage

Post-conversion pipeline

After converting new books, run the post-pipeline:

# Full pipeline: verify quality + refresh index.md + refresh figure registry + semantic index
python {REPO}/converter/post_convert.py

# Or individual steps:
python {REPO}/converter/post_convert.py --verify   # check quality only
python {REPO}/converter/post_convert.py --index    # semantic index only
python {REPO}/converter/post_convert.py --audit    # report index coverage (no backfill)

The pipeline does not extract figures — it only verifies page markers, refreshes index.md + figure_registry.json from whatever figures already exist, and (optionally) builds the semantic index. Figure extraction is never a batch post-step.

Missing page markers do not block indexing by design — EPUBs (reflowable) and some vector-glyph PDFs legitimately lack <!-- page N -->; the indexer handles this gracefully (page metadata = 0). After indexing, a coverage audit compares the markdown corpus against the index and auto-backfills any book that has markdown but no index rows, so every converted book ends up searchable even if a step was skipped.

Figures — do NOT batch-extract

No pre-extraction step. A whole-book batch figure dump is retired and should not be run on new books. When a note needs a figure, figure-remap extracts that single figure on demand with QC. See "Using figures in notes" below.

Chapter splitting (batch mode)

Chapter detection patterns (in priority order)

1. Chapter N / CHAPTER N — standard
2. Part N / PART N — with roman numerals (Part III)
3. Section N / SECTION N / Unit N
4. Numbered headings: 1 Introduction, 23 Shoulder (digit + title-case text)

Running header deduplication

Many textbooks repeat Chapter N Title as a running header on every page. Only count the first occurrence of each unique chapter number — track a seen_chapters set and skip duplicates.

Force-split rules

• No chapter breaks detected AND >200 pages → split every 30 pages (pages_0001-0030.md)
• Single output file >500K words → re-split (likely missed chapter breaks)