Agent skill

Doc Gen Unicode Diagnostic

by HKUDS in HKUDS/OpenSpace

Systematic document generation with unicode sanitization, engine fallback chain, and explicit error diagnosis

MITAuto-check passedDocuments & Office

Install Doc Gen Unicode Diagnostic

skills CLI
$ npx skills add HKUDS/OpenSpace --skill doc-gen-unicode-diagnostic -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install HKUDS/OpenSpace doc-gen-unicode-diagnostic --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/HKUDS/OpenSpace.git skills-src && mkdir -p .claude/skills && cp -r skills-src/benchmarks/gdpval/skills/document-gen-fallback-enhanced-enhanced-3d3a9a .claude/skills/doc-gen-unicode-diagnostic && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
doc-gen-unicode-diagnostic
GitHub stars
7.8k
Token cost
~2.7k tokens
SKILL.md length
607 words
Files
2
Skills in repo
199
Repo updated
First seen
Licence
MIT

At a glance

Systematic document generation with unicode sanitization, engine fallback chain, and explicit error diagnosis

  • Works in 7 steps: Pre-Flight Validation → Create Source Content with write_file → Create Sanitized Version for PDF… → …
  • Tasks that involve PDF
  • SKILL.md covers When to Use, ⚠️ Critical: Why This Skill…, Pre-Flight Validation (NEW -… and Core Technique, plus 10 more sections
  • Calls pandoc and apt-get

What it does

Doc Gen Unicode Diagnostic is an agent skill from HKUDS/OpenSpace. Systematic document generation with unicode sanitization, engine fallback chain, and explicit error diagnosis

Its SKILL.md is about 2.7k tokens, which your agent loads only when the skill is triggered. The skill folder holds 1 other file.

It sits in Documents & Office, covering PDF and LaTeX. It works with LaTeX. The repository describes itself as: "OpenSpace: The Skill Management Layer for AI Agents" -- https://open-space.cloud/. The licence is MIT.

When your agent uses it

  • Tasks that involve PDF
  • Tasks that involve LaTeX

Example prompts

  • “/doc-gen-unicode-diagnostic”

Requirements

  • Python 3

Workflow steps

7 steps, taken from the step headings in SKILL.md.

  1. Pre-Flight Validation
  2. Create Source Content with write_file
  3. Create Sanitized Version for PDF (MANDATORY)
  4. Convert to DOCX (from original, supports Unicode)
  5. Convert to PDF with Engine Fallback Chain (CRITICAL)
  6. Convert to HTML (from original)
  7. Verify All Outputs

What it can do on your machine

Read from SKILL.md and the folder at commit 3827781. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • pandoc
    • apt-get

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Doc Gen Unicode Diagnostic loads about 2.7k tokens when it runs. Until then it costs about 34 tokens; SKILL.md has 607 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~34
When it runs · the whole SKILL.md, loaded when a task matches
~2.7k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from HKUDS/OpenSpace at commit 3827781, republished under its MIT licence (© HKUDS). 607 words, ~2,652 tokens.

Download SKILL.mdSave it as .claude/skills/doc-gen-unicode-diagnostic/SKILL.md (or your agent's skills folder). This skill also uses 1 other file; get the full folder from GitHub.
name
doc-gen-unicode-diagnostic
description
Systematic document generation with unicode sanitization, engine fallback chain, and explicit error diagnosis

Document Generation with Diagnostic Workflow (Unicode-Safe)

When to Use

Use this skill when document generation tasks fail or return unclear errors, especially when:

  • Generating documents in multiple formats (.docx, .pdf, .html)
  • PDF generation fails with encoding/LaTeX errors
  • shell_agent returns "unknown error" without diagnostics
  • Documents contain special characters, symbols, or non-ASCII text
  • You need systematic error diagnosis rather than blind retries

⚠️ Critical: Why This Skill Exists

Recent executions show 43% effectiveness because agents:

  • Skip unicode sanitization before PDF conversion
  • Don't try xelatex engine (better Unicode support)
  • Don't capture stderr for proper diagnosis
  • Exhaust iterations on repeated failures without systematic troubleshooting

This skill fixes those gaps with mandatory steps.

Pre-Flight Validation (NEW - Required Step 0)

Before starting document generation, verify your toolchain:

run_shell
command: which pandoc && pandoc --version | head -3
run_shell
command: which pdflatex xelatex wkhtmltopdf 2>/dev/null || echo "Some engines missing"
run_shell
command: python3 -c "import sys; print(sys.version)"

If pandoc is missing, install it:

run_shell
command: apt-get update && apt-get install -y pandoc

For PDF support, install LaTeX engines:

run_shell
command: apt-get install -y texlive-latex-recommended texlive-fonts-recommended texlive-xetex

Core Technique

Manually split the workflow into observable, diagnostic steps:

  1. Pre-flight → Verify toolchain availability
  2. Content creation → Use write_file for markdown source (visible content)
  3. Unicode sanitization → MANDATORY for PDF: Create sanitized version
  4. Format conversion with fallback → Try engines in order, capture stderr
  5. Verification → Check outputs exist and validate content

Unicode & LaTeX Compatibility (MANDATORY for PDF)

PDF generation via LaTeX has limited Unicode support. Before PDF conversion, you MUST sanitize:

CharacterIssueSafe Replacement
— (em dash)LaTeX incompatibility--
– (en dash)LaTeX incompatibility-
" " (curly quotes)Encoding errors" " (straight)
' ' (curly apostrophe)Encoding errors' (straight)
… (ellipsis)May not render...
→ ← ↑ ↓ (arrows)LaTeX incompatibility-> <- ^ v
✓ ✗ (checkmarks)May not render[x] [ ]
★ ● (symbols)May not render* -
© ® ™Require packages(c) (r) (tm)
Non-ASCII (é, ñ, ü)Font-dependentKeep for xelatex, sanitize for pdflatex

DOCX and HTML handle Unicode natively - use original markdown for these formats.

Step-by-Step Workflow

Step 0: Pre-Flight Validation
run_shell
command: which pandoc || (apt-get update && apt-get install -y pandoc)
run_shell
command: which xelatex || echo "xelatex not available - will use fallback"
Step 1: Create Source Content with write_file
write_file
path: /tmp/document_source.md
content: |
  # Document Title
  
  ## Section 1
  Your content here with full Unicode support...
  
  ## Section 2
  Special chars: "quotes" — dashes … ellipsis ✓ checkmarks
Step 2: Create Sanitized Version for PDF (MANDATORY)

Option A: Manual sanitization with write_file

write_file
path: /tmp/document_source_sanitized.md
content: |
  # Document Title
  
  ## Section 1
  Your content here...
  
  ## Section 2
  Special chars: "quotes" -- dashes ... ellipsis [x] checkmarks

Option B: Automated sanitization script

First create the script:

write_file
path: /tmp/sanitize_for_pdf.sh
content: |
  #!/bin/bash
  INPUT="$1"
  OUTPUT="${2:-${1%.md}_sanitized.md}"
  sed -e 's/—/--/g' -e 's/–/-/g' \
      -e 's/"([^"]*)"/"\1"/g' -e "s/'([^']*)'/\`\1\`/g" \
      -e 's/…/.../g' -e 's/→/->/g' -e 's/←/<-/g' \
      -e 's/✓/[x]/g' -e 's/✗/[ ]/g' \
      -e 's/©/(c)/g' -e 's/®/(r)/g' -e 's/™/(tm)/g' \
      "$INPUT" > "$OUTPUT"
  echo "Sanitized: $INPUT -> $OUTPUT"
run_shell
command: chmod +x /tmp/sanitize_for_pdf.sh && /tmp/sanitize_for_pdf.sh /tmp/document_source.md /tmp/document_source_sanitized.md
Step 3: Convert to DOCX (from original, supports Unicode)
run_shell
command: pandoc /tmp/document_source.md -o output.docx 2>&1

Capture stderr with 2>&1 to see actual errors (not "unknown error").

Step 4: Convert to PDF with Engine Fallback Chain (CRITICAL)

Try engines in order: xelatex (best Unicode) → pdflatex → wkhtmltopdf

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=xelatex 2>&1

If xelatex fails, try pdflatex:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=pdflatex 2>&1

If pdflatex fails, try wkhtmltopdf:

run_shell
command: pandoc /tmp/document_source_sanitized.md -o output.pdf --pdf-engine=wkhtmltopdf 2>&1

If ALL engines fail, diagnose with:

run_shell
command: file /tmp/document_source_sanitized.md && head -20 /tmp/document_source_sanitized.md
Show full SKILL.md (244 more words)Show less
Step 5: Convert to HTML (from original)
run_shell
command: pandoc /tmp/document_source.md -o output.html 2>&1
Step 6: Verify All Outputs
run_shell
command: ls -lh output.* && file output.*
run_shell
command: [ -f output.pdf ] && echo "PDF created: $(wc -c < output.pdf) bytes" || echo "PDF MISSING"

Complete Example

markdown
# Generate Client Report in Multiple Formats

## Step 0: Pre-flight
run_shell
command: which pandoc xelatex || echo "Checking toolchain..."

## Step 1: Write markdown source
write_file
path: /tmp/client_report.md
content: |
  # Client Investment Report
  
  ## Executive Summary
  Portfolio performance shows strong returns — up 15% this quarter...
  
  ## Risk Analysis
  Key metrics: "Sharpe ratio" ✓ passed … continuing analysis

## Step 2: Sanitize for PDF (MANDATORY)
write_file
path: /tmp/client_report_sanitized.md
content: |
  # Client Investment Report
  
  ## Executive Summary
  Portfolio performance shows strong returns -- up 15% this quarter...
  
  ## Risk Analysis
  Key metrics: "Sharpe ratio" [x] passed ... continuing analysis

## Step 3: Convert to DOCX (original unicode OK)
run_shell
command: pandoc /tmp/client_report.md -o client_report.docx 2>&1

## Step 4: Convert to PDF (sanitized, xelatex first)
run_shell
command: pandoc /tmp/client_report_sanitized.md -o client_report.pdf --pdf-engine=xelatex 2>&1

## Step 5: Convert to HTML (original unicode OK)
run_shell
command: pandoc /tmp/client_report.md -o client_report.html 2>&1

## Step 6: Verify
run_shell
command: ls -lh client_report.* && file client_report.*

Error Diagnosis Decision Tree

When a conversion fails, capture stderr (2>&1) and diagnose:

If error contains "xelatex not found" or "LaTeX error":
  → Try next engine: --pdf-engine=pdflatex or --pdf-engine=wkhtmltopdf

If error contains "encoding" or "UTF-8":
  → Unicode not properly sanitized; re-check Step 2
  → Add -f markdown+utf8 to pandoc command

If error contains "template" or "class":
  → LaTeX template issue; try --pdf-engine=wkhtmltopdf

If error is "unknown error" (no stderr captured):
  → Re-run with 2>&1 to capture actual error message
  → Check if pandoc is installed: which pandoc

If wkhtmltopdf fails:
  → Install: apt-get install wkhtmltopdf
  → Or use Python alternative: reportlab or fpdf2

Alternative: Python PDF Generation (When pandoc Fails)

If all pandoc engines fail, use Python libraries directly:

Using fpdf2:

run_shell
command: python3 -c "
from fpdf import FPDF
pdf = FPDF()
pdf.add_page()
pdf.set_font('Arial', '', 12)
pdf.cell(0, 10, 'Document Title')
pdf.output('output.pdf')
"

Using reportlab:

run_shell
command: python3 -c "
from reportlab.lib.pagesizes import letter
from reportlab.pdfgen import canvas
c = canvas.Canvas('output.pdf', pagesize=letter)
c.drawString(100, 750, 'Document Title')
c.save()
"

Common pandoc Commands Reference

bash
# Markdown to Word (Unicode-safe)
pandoc input.md -o output.docx

# Markdown to PDF with xelatex (BEST for Unicode)
pandoc input.md -o output.pdf --pdf-engine=xelatex

# Markdown to PDF with pdflatex (requires sanitization)
pandoc input.md -o output.pdf --pdf-engine=pdflatex

# Markdown to PDF with wkhtmltopdf (HTML-based, good fallback)
pandoc input.md -o output.pdf --pdf-engine=wkhtmltopdf

# Markdown to HTML
pandoc input.md -o output.html

# With metadata
pandoc input.md -o output.pdf --metadata title="Document Title"

# Force UTF-8 encoding
pandoc -f markdown+utf8 input.md -o output.pdf

Troubleshooting Quick Reference

SymptomLikely CauseSolution
"xelatex not found"Missing LaTeX engineapt-get install texlive-xetex or try --pdf-engine=wkhtmltopdf
"LaTeX error: encoding"Unicode in sourceUse sanitized markdown for PDF
"unknown error" (pandoc)stderr not capturedRe-run with 2>&1 to see real error
PDF missing after conversionAll engines failedTry Python (fpdf2/reportlab) as fallback
DOCX has garbled textEncoding issueAdd -f markdown+utf8 to pandoc command
HTML renders but PDF failsLaTeX-specific issuewkhtmltopdf engine usually works

Verification Checklist

Before marking task complete, verify:

  • Pre-flight: pandoc installed and accessible
  • Source markdown created with write_file
  • Sanitized version created for PDF conversion
  • DOCX generated from original (unicode preserved)
  • PDF generated with xelatex (or fallback engine documented)
  • All output files exist: ls -lh output.*
  • File types verified: file output.*
  • Content validated (spot-check with read_file if applicable)

When to Use shell_agent Instead

After successfully completing this manual workflow:

  • For simple DOCX-only tasks (no PDF needed)
  • When toolchain is verified working
  • For repetitive tasks with known-good content

For documents with Unicode content requiring PDF, always use this manual workflow.

  • spreadsheet-direct-python: For Excel/CSV generation with Python
  • pdf-verification-cli: For verifying PDF page count and content after creation

© HKUDS, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file in benchmarks/gdpval/skills/document-gen-fallback-enhanced-enhanced-3d3a9a of HKUDS/OpenSpace.

  • SKILL.md
  • .skill_id

Open the folder on GitHubat commit 3827781

Compare with similar skills

Doc Gen Unicode Diagnostic next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Doc Gen Unicode Diagnostic compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Doc Gen Unicode Diagnostic this skillHKUDS/OpenSpace7.8k—~2.7kAutomated safety check: PassMIT
AI Review SkillNeuroDong/Ai-Review626—~2.5kAutomated safety check: PassMIT
MineruNebutra/MinerU-Skill122—~504Automated safety check: PassMIT
Lecture To Mdysyecust/lecture-to-notes273—~3.9kAutomated safety check: PassCustom licence
Paper CompileAI4Scientist/nano-scientist1285 repos~2.5kAutomated safety check: NotesNone
Pdf2texCalix-L/awesome-latex-skills181—~1.4kAutomated safety check: PassMIT

Similar skills

  • AI Review Skill

    NeuroDong/Ai-Review

    Generates structured AI paper reviews (SoT style) for LaTeX, PDF, and Word manuscripts.

    626 GitHub stars~2.5k tokensUpdated 1 mo ago
    Documents & OfficeAuto-check passed
  • Mineru

    Nebutra/MinerU-Skill

    An AI-Native skill for parsing PDF / Office / image files into Markdown with MinerU — a fast, zero-config document parser for AI agents.

    122 GitHub stars~504 tokensUpdated 15 days ago
    Documents & OfficeAuto-check passed
  • Lecture To Md

    ysyecust/lecture-to-notes

    把课堂视频(本地或 B 站/YouTube)、文字稿、课件三者(任意组合)整理成一份详细的中文 Markdown 课堂笔记,输出按课程标题命名的 {titlename}.md(首行为 文档标题)+ 相对路径图片。Markdown 工作流,与上游 lecture-to-notes 的 LaTeX/PDF 输出并行存在;上游 skill 完全不动。触发词:markdown 笔记、md 笔记、视频转…

    273 GitHub stars~3.9k tokensUpdated 6 days ago
    Documents & OfficeAuto-check passed
  • Paper Compile

    AI4Scientist/nano-scientist

    Compile LaTeX paper to PDF, fix errors, and verify output. An agent skill from AI4Scientist/nano-scientist.

    128 GitHub starsUsed in 5 repos~2.5k tokens
    Documents & OfficeAuto-check: notes
  • Pdf2tex

    Calix-L/awesome-latex-skills

    Reconstruct editable LaTeX from PDF content using page-aware extraction and visual comparison.

    181 GitHub stars~1.4k tokensUpdated 2 days ago
    Documents & OfficeAuto-check passed
  • 中国软件著作权申请材料生成工具。申请表直接输出 Markdown 提交,源程序/用户手册/设计说明书三份生成 LaTeX 并编译为 PDF。自动分析项目代码,生成四份材料(前后各30页共60页源程序、含页眉页脚的用户手册和设计说明书、Markdown…

    183 GitHub stars~2.8k tokensUpdated 2 days ago
    Documents & OfficeAuto-check: notes

More from HKUDS/OpenSpace

All 199 skills in this repo
  • Walks through producing a master audio track plus stems in Python, from checking a reference file and timing sections by BPM to effects, a zip archive and final verification.

    7.8k GitHub stars~2.9k tokensUpdated 1 mo ago
    Auto-check passed
  • Handle cascading data retrieval tool failures by falling back to embedded knowledge generation

    7.8k GitHub stars~765 tokensUpdated 1 mo ago
    Auto-check passed
  • Gives an agent a workaround when its code-execution sandbox keeps failing: save the Python script to a file and run it through the shell instead.

    7.8k GitHub stars~588 tokensUpdated 1 mo ago
    Auto-check passed
  • A recovery routine for agents whose sandboxed code runner keeps failing: save the Python script to disk, then run it through the shell and read the output.

    7.8k GitHub stars~652 tokensUpdated 1 mo ago
    Auto-check passed
  • Fallback ladder for failed sandboxed code runs, plus the habit of fixing the working directory first so generated files land in the right place.

    7.8k GitHub stars~1.1k tokensUpdated 1 mo ago
    Auto-check passed
  • Fallback workflow for executing Python code when executecodesandbox fails repeatedly

    7.8k GitHub stars~1.1k tokensUpdated 1 mo ago
    Auto-check passed

Works with

Questions about Doc Gen Unicode Diagnostic

What does Doc Gen Unicode Diagnostic do?

Systematic document generation with unicode sanitization, engine fallback chain, and explicit error diagnosis. Doc Gen Unicode Diagnostic is an agent skill from HKUDS/OpenSpace.

When should I use Doc Gen Unicode Diagnostic?

Doc Gen Unicode Diagnostic fits situations like: tasks that involve PDF; tasks that involve LaTeX.

How do I install Doc Gen Unicode Diagnostic in Claude Code?

Run `npx skills add HKUDS/OpenSpace --skill doc-gen-unicode-diagnostic -a claude-code`. Or copy the skill folder (benchmarks/gdpval/skills/document-gen-fallback-enhanced-enhanced-3d3a9a in HKUDS/OpenSpace) into .claude/skills/doc-gen-unicode-diagnostic in your project. Claude Code loads it when a task matches its description.

How do I install Doc Gen Unicode Diagnostic in Codex?

Run `npx skills add HKUDS/OpenSpace --skill doc-gen-unicode-diagnostic -a codex`. Or copy the skill folder (benchmarks/gdpval/skills/document-gen-fallback-enhanced-enhanced-3d3a9a in HKUDS/OpenSpace) into .agents/skills/doc-gen-unicode-diagnostic in your project. Codex loads it when a task matches its description.

Can I use Doc Gen Unicode Diagnostic in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add HKUDS/OpenSpace --skill doc-gen-unicode-diagnostic -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/doc-gen-unicode-diagnostic, .gemini/skills/doc-gen-unicode-diagnostic, .github/skills/doc-gen-unicode-diagnostic and .opencode/skills/doc-gen-unicode-diagnostic in your project.

What does Doc Gen Unicode Diagnostic need to run?

Going by SKILL.md and its folder, Doc Gen Unicode Diagnostic needs the command-line tools its instructions call (pandoc and apt-get). Our summary lists: Python 3.

Does Doc Gen Unicode Diagnostic access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is Doc Gen Unicode Diagnostic safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Doc Gen Unicode Diagnostic use?

Doc Gen Unicode Diagnostic is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Doc Gen Unicode Diagnostic use?

About 2.7k tokens (SKILL.md is roughly 11k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Doc Gen Unicode Diagnostic?

Skills that share tags, products or a category with Doc Gen Unicode Diagnostic: AI Review Skill (NeuroDong/Ai-Review, 626 stars), Mineru (Nebutra/MinerU-Skill, 122 stars), Lecture To Md (ysyecust/lecture-to-notes, 273 stars) and Paper Compile (AI4Scientist/nano-scientist, 128 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Doc Gen Unicode Diagnostic?

HKUDS (a GitHub organization) maintains it in HKUDS/OpenSpace, which has 7,750 GitHub stars. The repository holds 199 skills in this directory. The repository was last updated on August 12, 2026.

Source: HKUDS/OpenSpace on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.