HTML Conformance

Use this skill when asked to advance Panache's HTML conformance, unblock a regression that involves raw HTML attributes (issue #263 and its descendants), or pick "the next best phase" of the HTML lift.

What this skill is NOT

• Not a chase for the conformance pass-rate. The pass-rate is a metric, not a goal. A passing case can still hide a wrong CST if the projector compensates (re-parses bytes, walks text instead of children, makes context-dependent decisions at projection time). When that happens, the projector silently absorbs structural bugs the CST should have surfaced.
• Not a place to add projector logic that papers over CST gaps. The projector at crates/panache-parser/src/pandoc_ast.rs is a test-only diagnostic — its job is to reveal CST shape problems by diffing against pandoc-native. Putting logic there to make a test pass while the CST stays wrong destroys the diagnostic value. The consumers of structural HTML decisions (linter, salsa, LSP, formatter) read the CST, not the projector output.
• Not "make it look like pandoc's output text." The objective is for our CST to encode the same structural decisions pandoc encodes in its AST — Plain vs Para, Div vs RawBlock, RawBlock vs RawInline, matched-pair vs single emit, etc. — so reading the CST gives you the same answers reading pandoc's AST would.

If a session's diff is mostly in pandoc_ast.rs (other than removing existing compensation), that's a smell. The fix probably belongs in the parser.

Scope boundaries

• Target is HTML-block + raw-HTML parsing under Flavor::Pandoc. Block-level: crates/panache-parser/src/parser/blocks/html_blocks.rs
  • block_dispatcher.rs. Inline-level: crates/panache-parser/src/parser/inlines/inline_html.rs. Projection: crates/panache-parser/src/pandoc_ast.rs. Salsa indexer: src/salsa.rs.
• Flavor::Pandoc only. CommonMark dialect must stay byte-identical in CST and pandoc-native projection. Dialect::CommonMark keeps the opaque HTML_BLOCK shape; lifts are gated on Dialect::Pandoc.
• Pandoc-native (pandoc -f markdown -t native) is the behavioral reference. Existing parser fixtures and projector output are not the reference — when they disagree with pandoc-native, fix toward pandoc-native.
• This is a long-horizon effort (5 phases — see "Phased plan" below). Each session moves at most one phase forward; no sweeping rewrites in a single go.
• Reuses the existing pandoc-conformance harness verbatim. New cases live in crates/panache-parser/tests/fixtures/pandoc-conformance/corpus/<NNNN>-<section>-<slug>/ with section prefix html-block (block-level) or html-inline (inline-level). The pandoc allowlist (crates/panache-parser/tests/pandoc/allowlist.txt) gets new section header comments # html-block / # html-inline.
• Out-of-scope and deferred to crates/panache-parser/tests/pandoc/blocked.txt: markdown="1" / markdown="0" (Ext_markdown_attribute, default off in pandoc-flavored markdown); <a id> / <a name> legacy anchor lift (pandoc does NOT lift these in default markdown — treat as opaque RawInline); malformed/unbalanced tags (fall back to opaque HTML_BLOCK / INLINE_HTML).

Follow the Pandoc conformance, parser, and formatter invariants in the repository's root AGENTS.md.

Phased plan

The HTML lift is bounded by 5 phases. Pick one (or part of one) per session. Latest phase status lives in RECAP.md.

Phase 1 — Block-level <div> lift (Pandoc dialect). Parser emits HTML_BLOCK_DIV for matched <div ...>...</div>; projector consumes it and emits Block::Div(attrs, blocks); salsa indexer extracts id from the open tag and registers it in crossref_declarations. Unblocks issue #263.

Phase 2 — Inline <span> lift (Pandoc dialect). Mirrors Phase 1 on the inline side. Coordinate with pandoc-ir-migrate Phase 1 — <span> is already a ConstructKind::PandocOpaque event in inline_ir.rs, so the lift must not double-handle the byte range.

Phase 3 — Sectioning (<section>, <article>, <aside>, <nav>) and verbatim (<pre>, <style>, <script>, <textarea>) parity. Negative-space work: confirm pandoc-native keeps these as RawBlock "html" with no markdown parsing inside, and pin that behavior in the corpus. No CST shape change expected; verify the projector emits exactly what pandoc does.

Phase 4 — Comments, processing instructions, declarations, CDATA projection. Pin RawBlock "html" / RawInline "html" for each case. CST already correct — this is mostly seeding the corpus and verifying projection output.

Phase 5 — markdown_in_html_blocks interaction edge cases: nested div-in-div; div-around-list; div-with-blank-lines (Plain vs Para promotion); block-tag-on-same-line shapes (<div>foo</div>); div-as-list-item-content. The hardest phase — expect parser-side fixes to support pandoc's recursive parse.

Key files

Parser

• crates/panache-parser/src/parser/blocks/html_blocks.rs — block HTML state machine. try_parse_html_block_start recognizes the 7 CommonMark types; parse_html_block walks lines and emits the CST. Phase 1 adds a use_div_kind: bool parameter that retags the wrapper from HTML_BLOCK to HTML_BLOCK_DIV when a matched <div>...</div> is recognized under Pandoc.
• crates/panache-parser/src/parser/block_dispatcher.rs — calls try_parse_html_block_start and parse_html_block. Phase 1 adds the matched-</div> pre-scan and passes the result down.
• crates/panache-parser/src/parser/inlines/inline_html.rs — inline raw HTML. Phase 2's <span> lift adds an INLINE_HTML_SPAN shape gated on Pandoc dialect.
• crates/panache-parser/src/syntax/kind.rs — new SyntaxKinds. Phase 1 adds HTML_BLOCK_DIV. Phase 2 adds INLINE_HTML_SPAN.

Projector

• crates/panache-parser/src/pandoc_ast.rs — projector. Phase 1 adds an HTML_BLOCK_DIV match arm in block_from / collect_block that walks the structural CST instead of re-tokenizing bytes. The legacy try_div_html_block byte-level re-tokenizer can stay as a fallback for HTML_BLOCK (when the parser hasn't lifted, e.g. malformed input under Pandoc), or be deleted once all matched divs are lifted at parse time.
• parse_html_attrs (already in pandoc_ast.rs) — reused to extract attributes from the open tag's verbatim text.

Salsa / linter consumers

• src/salsa.rs — anchor index. Phase 1 adds a walk for HTML_BLOCK_DIV that reads the first HTML_BLOCK_TAG child, extracts attributes via parse_html_attrs, and registers the id in crossref_declarations. Phase 2 adds the same for INLINE_HTML_SPAN.
• src/linter/rules/undefined_anchor.rs — consumer of the index. No code change expected, just integration tests confirming that <div id> no longer produces false positives.

Formatter

• crates/panache-formatter/src/formatter/core.rs — has multiple match arms keyed on SyntaxKind::HTML_BLOCK. Phase 1 needs to also accept HTML_BLOCK_DIV at each site (text emission is identical). Likewise crates/panache-formatter/src/utils.rs, formatter/lists.rs, and directives.rs for any HTML_BLOCK match.

Tests / fixtures

• crates/panache-parser/tests/fixtures/pandoc-conformance/corpus/ — corpus, section prefix html-block / html-inline.
• crates/panache-parser/tests/pandoc/allowlist.txt — new # html-block / # html-inline section comments.
• crates/panache-parser/tests/pandoc/blocked.txt — deferrals.
• crates/panache-parser/tests/fixtures/cases/ — paired parser golden fixtures (CommonMark vs Pandoc) when CST shape diverges by dialect.
• tests/fixtures/cases/ — formatter golden fixtures when Phase N produces a new round-trip behavior. Required when CST shape changes — <div> must format back as <div>, not :::.

Losslessness invariant + Phase 1 precedent

The CST must be byte-equal to the input. The right way to expose HTML attributes structurally is to tokenize the existing source bytes at finer granularity, not to add synthetic tokens. Phase 1 demonstrates the pattern for <div>:

The open-tag bytes <div id="x" class="y"> previously lived in a single TEXT token. Phase 1 splits them into `TEXT("<div") + WHITESPACE + HTML_ATTRS{TEXT("id="x" class="y"")}

• TEXT(">"). Source bytes are unchanged — the structural node HTML_ATTRSjust *groups* the existing attribute bytes soAttributeNode::cast can recognize them. This mirrors how fenced divs already work (DIV_INFOgroups the{#id .class}bytes after:::`).

For Phase 2 (<span>), do the same thing inline: <span id="x">...</span> should tokenize the open tag's attribute region into an HTML_ATTRS node so the same AttributeNode walk finds it automatically.

Phase 1 established the reusable pattern: retag matched Pandoc wrappers, tokenize their existing attribute bytes as HTML_ATTRS, and let the standard AttributeNode walk expose them. Inner block content and multi-line open tags remain Phase 5 work. Phase 2 mirrors this shape for matched <span> wrappers.

Failure buckets

Every failing HTML conformance case is one of:

• Projector gap — parser produces structural HTML_BLOCK_DIV (or HTML_BLOCK for non-divs) but the projector emits the wrong pandoc-native shape (missing attribute, wrong block sequencing, wrong RawBlock vs Plain split). Fix in pandoc_ast.rs.
• Parser-shape gap — parser produces opaque HTML_BLOCK for a construct that should lift, or vice versa. Fix in html_blocks.rs / inline_html.rs. Add a paired parser fixture pinning the CST shape under both dialects when behavior diverges.
• Salsa-index gap — projector and parser are correct but the anchor index doesn't see the id. Fix in src/salsa.rs.
• Flavor / extension gap — pandoc-native enables/disables the lift based on an extension (Ext_native_divs, Ext_native_spans, Ext_markdown_in_html_blocks); panache's default disagrees. Tighten in crates/panache-parser/src/options.rs::pandoc_defaults().
• Genuine missing feature — pandoc construct not modeled (e.g. <aside> with markdown="1"). Add to blocked.txt with reason unless the construct is a clear next-phase target.

Workflow (per session)

1. Read RECAP.md for current phase, deferred targets, traps. If the user named a target, prefer it.

2. Establish the test baseline:

   cargo test -p panache-parser --test pandoc pandoc_allowlist
   cargo test -p panache-parser --test commonmark commonmark_allowlist
   cargo test --workspace --no-fail-fast 2>&1 | grep -E "^test " | grep "FAILED" | sort -u

   Save the failing-test set; "no regression" means a strict subset.

3. Regenerate the conformance report (skip if last entry is stale by less than 24h):

   cargo test -p panache-parser --test pandoc pandoc_full_report \
     -- --ignored --nocapture

   Look at crates/panache-parser/tests/pandoc/report.txt for the # html-block and # html-inline slices.

4. Pick a target:

   • A handful of failing html-* cases sharing a likely root cause (e.g. all div-in-list cases failing → one parser-shape fix unlocks several).
   • A small corpus expansion to cover an unmodeled construct, if phase warrants.

5. Probe the case(s):

   pandoc <case>/input.md -f markdown -t native
   pandoc <case>/input.md -f commonmark -t native   # if dialect-divergent

   For batch triage, drop a throwaway probe test in crates/panache-parser/tests/pandoc.rs (the same template as pandoc-conformance.md describes) and delete it before finishing the session.

6. Classify into a failure bucket, then apply the smallest fix. Verify the change is parser-side rather than projector-side only when downstream consumers (salsa, LSP, formatter) need the structure — projector-only fixes are fine when the structure is already in the CST.

7. Add fixtures before allowlisting:

   • Parser fixture (paired CommonMark/Pandoc when behavior diverges) under crates/panache-parser/tests/fixtures/cases/. Required when adding a new CST shape.
   • Formatter fixture under tests/fixtures/cases/ when the change produces a new block sequence or different idempotency behavior. Pandoc is the default flavor — no panache.toml needed.

8. Verify each new corpus case appears in the regenerated report.txt before adding to the allowlist:

   grep -E '^(N1|N2|N3)$' \
     crates/panache-parser/tests/pandoc/report.txt

   Each id must show up in the passing list. Add under the # html-block / # html-inline section header.

9. Run guardrails:

   cargo test -p panache-parser --test pandoc pandoc_allowlist
   cargo test -p panache-parser --test commonmark commonmark_allowlist
   cargo test -p panache-parser
   cargo test --workspace
   cargo clippy --workspace --all-targets --all-features -- -D warnings
   cargo fmt -- --check
   cargo test -p panache-parser --test pandoc pandoc_full_report \
     -- --ignored --nocapture

   The CommonMark allowlist must stay green; CST losslessness (parser-crate's tree-text-equals-input check) must stay green.

10. Update RECAP.md with the session outcome: phase touched, pass-rate before/after for the html-block / html-inline slices, files changed, new traps, ranked next targets.