Po Translate
natsukium/dotfiles
Orchestrate English→Japanese translation of po/ja.po — classify, delegate translation/review to subagents, iterate until clean
Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report.
Install Benchmark Translate
$ npx skills add shapeshift/web --skill benchmark-translate -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install shapeshift/web benchmark-translate --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/shapeshift/web.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.claude/skills/benchmark-translate .claude/skills/benchmark-translate && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "benchmark-translate" agent skill from https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translate into .claude/skills/benchmark-translate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-translate", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translateType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add shapeshift/web --skill benchmark-translate -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install shapeshift/web benchmark-translate --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/shapeshift/web.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.claude/skills/benchmark-translate .agents/skills/benchmark-translate && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "benchmark-translate" agent skill from https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translate into .agents/skills/benchmark-translate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-translate", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add shapeshift/web --skill benchmark-translate -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install shapeshift/web benchmark-translate --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/shapeshift/web.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.claude/skills/benchmark-translate .cursor/skills/benchmark-translate && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "benchmark-translate" agent skill from https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translate into .cursor/skills/benchmark-translate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-translate", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/shapeshift/web.git --path .claude/skills/benchmark-translate--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add shapeshift/web --skill benchmark-translate -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install shapeshift/web benchmark-translate --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/shapeshift/web.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.claude/skills/benchmark-translate .gemini/skills/benchmark-translate && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "benchmark-translate" agent skill from https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translate into .gemini/skills/benchmark-translate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-translate", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install shapeshift/web benchmark-translateInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add shapeshift/web --skill benchmark-translate -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/shapeshift/web.git skills-src && mkdir -p .github/skills && cp -r skills-src/.claude/skills/benchmark-translate .github/skills/benchmark-translate && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "benchmark-translate" agent skill from https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translate into .github/skills/benchmark-translate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-translate", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add shapeshift/web --skill benchmark-translate -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install shapeshift/web benchmark-translate --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/shapeshift/web.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.claude/skills/benchmark-translate .opencode/skills/benchmark-translate && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "benchmark-translate" agent skill from https://github.com/shapeshift/web/tree/develop/.claude/skills/benchmark-translate into .opencode/skills/benchmark-translate/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "benchmark-translate", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
Facts
- Skill name
benchmark-translate- GitHub stars
- 206
- Token cost
- ~1.6k tokens
- SKILL.md length
- 526 words
- Files
- 5 (incl. scripts)
- Skills in repo
- 7
- Repo updated
- First seen
- Licence
- MIT
At a glance
Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report.
- Works in 7 steps: Select Keys → Setup → Translate → …
- Tasks that involve Translation
- SKILL.md covers Data Artifacts and Pipeline (7 Steps)
- Runs JavaScript scripts from its folder; calls node and git
What it does
Benchmark Translate is an agent skill from shapeshift/web. Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report. Invoke with /benchmark-translate.
Its SKILL.md is about 1.6k tokens, which your agent loads only when the skill is triggered. The skill folder holds 5 other files, including scripts (for example `scripts/compile-report.js`, `scripts/restore.js` and `scripts/select-keys.js`).
It sits in Writing & Content, covering Translation, Subagents and LLM evaluation. The licence is MIT.
When your agent uses it
- Tasks that involve Translation
- Tasks that involve Subagents
- Tasks that involve LLM evaluation
Example prompts
- “/benchmark-translate”
Requirements
- Node.js
- Pre-approved tools (allowed-tools): Read, Write, Edit, Grep, Glob, Bash(node *), Bash(git checkout*), Bash(git diff*), Bash(git status*), Bash(git rev-parse*), Task, Skill, AskUserQuestion
Workflow steps
7 steps, taken from the step headings in SKILL.md.
What it can do on your machine
Read from SKILL.md and the folder at commit 45096d2. It shows what the files ask for, not the result of running them.
Tool permissions
Pre-approves these tools, so the agent can use them without asking each time:
ReadWriteEditGrepGlobBash(node *)Bash(git checkout*)Bash(git diff*)Bash(git status*)Bash(git rev-parse*)
…and 3 more on the same allowed-tools line.
From allowed-tools in the SKILL.md frontmatter.
Runs code
Ships 4 files in scripts/ (JavaScript), which the agent can run.
Shell commands in SKILL.md call:
nodegit
From the folder's file list and the shell code blocks in SKILL.md.
Network
No URLs in SKILL.md. Its commands use git, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Credentials
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Context cost
Benchmark Translate loads about 1.6k tokens when it runs. Until then it costs about 58 tokens; SKILL.md has 526 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
Safety
Auto-check passed
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.
SKILL.md
The full file from shapeshift/web at commit 45096d2, republished under its MIT licence (© shapeshift). 526 words, ~1,550 tokens.
Download SKILL.mdSave it as
.claude/skills/benchmark-translate/SKILL.md (or your agent's skills folder). This skill also uses 4 other files; get the full folder from GitHub.- name
- benchmark-translate
- description
- Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report. Invoke with /benchmark-translate.
- allowed-tools
- Read, Write, Edit, Grep, Glob, Bash(node *), Bash(git checkout*), Bash(git diff*), Bash(git status*), Bash(git rev-parse*), Task, Skill, AskUserQuestion
Translation Quality Benchmark
Measures the quality of the /translate skill by comparing its output against existing human translations. Uses stratified key selection with a fixed/rotating split, LLM judges, and programmatic validation to produce a comprehensive quality report with regression tracking across all 9 supported locales.
Data Artifacts
All benchmark data lives in scripts/translations/benchmark/ (gitignored):
| File | Purpose |
|---|---|
testKeys.json | Selected test keys with categories and fixed flag |
coreKeys.json | Persistent core key set (stable across runs) |
ground-truth.json | Captured human translations before removal |
report.json | Latest benchmark report (becomes baseline on next run) |
baseline.json | Previous report (auto-copied by setup.js) |
Pipeline (7 Steps)
Step 1: Select Keys
node .claude/skills/benchmark-translate/scripts/select-keys.js [--count N] [--core N]Selects N keys (default 150) stratified across 6 categories: glossary-term, financial-error, single-word, interpolation, defi-jargon, general. Validates all selected keys exist in en + all 9 locales.
Fixed/rotating split:
--core N(default 100): Number of fixed core keys for stable regression tracking--count N(default 150): Total keys (core + rotating)- If
coreKeys.jsonexists: loads it, validates keys still exist in all locales, tops up if needed - If
coreKeys.jsondoesn't exist: selects core keys via stratified sampling and saves them - Remaining keys (default 50) are randomly selected as rotating keys from the non-core pool
- Each entry in
testKeys.jsonhas"fixed": true(core) or"fixed": false(rotating)
Outputs scripts/translations/benchmark/testKeys.json.
Step 2: Setup
node .claude/skills/benchmark-translate/scripts/setup.js- If
report.jsonexists from a previous run, copies it tobaseline.json - Reads
testKeys.json, captures ground truth translations for all 9 locales - Writes
ground-truth.json - Removes test keys from locale files so
/translatecan regenerate them
Step 3: Translate
Invoke the /translate skill using the Skill tool. This regenerates the removed keys through the full translate-review-refine pipeline.
Show full SKILL.md (262 more words)Show less
Step 4: Judge (Sub-Agents)
Launch 9 sub-agents in 3 waves of 3 (matching /translate's wave structure) using the Task tool. Each sub-agent receives the locale info, all key triplets, and glossary terms.
Wave 1: de, es, fr Wave 2: pt, ru, tr Wave 3: ja, uk, zh
For each locale, use this prompt:
You are an expert multilingual localization quality assessor for a cryptocurrency/DeFi application.
Rate translations from English into {LANGUAGE_NAME} on a 1-5 scale.
1 = Wrong/misleading meaning
2 = Significant issues (wrong register, missing nuance)
3 = Acceptable but could be more natural
4 = Good, natural, accurate
5 = Excellent, indistinguishable from professional native translation
Check: meaning preservation, naturalness, register ({REGISTER}), UI conciseness,
glossary compliance (these stay English: {NEVER_TRANSLATE_TERMS}),
placeholder integrity (%{...} preserved), DeFi terminology conventions.
Rate each translation INDEPENDENTLY. Community translations can contain errors.
Input: JSON array of {key, english, human, skill}
{ITEMS_JSON}
Output: Return ONLY a JSON array of objects with these exact fields:
{key, humanScore, skillScore, humanJustification, skillJustification, preferenceNote}
Scores must be integers 1-5. Justifications should be 1-2 sentences. preferenceNote should say which is better and why, or "tie" if equal.Locale info for prompt substitution:
| Locale | Language | Register |
|---|---|---|
de | German | Formal (Sie) |
es | Spanish | Informal (tú) |
fr | French | Formal (vous) |
ja | Japanese | Polite (です/ます) |
pt | Portuguese | Informal (você) |
ru | Russian | Formal (вы) |
tr | Turkish | Formal (siz) |
uk | Ukrainian | Formal (ви) |
zh | Chinese (Simplified) | Neutral/formal |
Building the items array for each locale:
- Read
scripts/translations/benchmark/ground-truth.json - Read the current (post-translate)
src/assets/translations/{locale}/main.json - For each test key, build:
{ key: dottedPath, english: groundTruth.english[key], human: groundTruth.groundTruth[locale][key], skill: getValueFromLocaleFile(key) }
Getting never-translate terms: Read src/assets/translations/glossary.json, collect all keys where value is null (excluding _meta).
Each sub-agent must write its output to /tmp/{locale}-judge-scores.json. Parse the JSON array from the sub-agent's response and write it to that path.
Step 5: Compile Report
node .claude/skills/benchmark-translate/scripts/compile-report.jsLoads judge scores from /tmp/{locale}-judge-scores.json, runs programmatic validation (including Cyrillic script check for ru/uk), computes summary stats, and writes scripts/translations/benchmark/report.json. If baseline.json exists, includes regression deltas. Report includes coreSummary and rotatingSummary alongside the overall summary.
Step 6: Restore
node .claude/skills/benchmark-translate/scripts/restore.jsRestores locale files via git checkout --, verifies no diff remains.
Step 7: Present Results
Read the compile output (printed to stdout) and present to the user:
- Overall score summary with baseline regression (if available)
- Core vs rotating stats (divergence suggests overfitting)
- Notable improvements/regressions
- Per-locale and per-category highlights
- Any items needing attention (low scores, validation failures, glossary issues)
© shapeshift, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Files
SKILL.md and 4 other files (scripts) in .claude/skills/benchmark-translate of shapeshift/web.
- SKILL.md
- scripts/compile-report.js
- scripts/restore.js
- scripts/select-keys.js
- scripts/setup.js
Open the folder on GitHubat commit 45096d2
Compare with similar skills
Benchmark Translate next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Benchmark Translate this skillshapeshift/web | 206 | — | ~1.6k | Automated safety check: Pass | MIT | |
| Po Translatenatsukium/dotfiles | 106 | — | ~1.4k | Automated safety check: Pass | CC0-1.0 | |
| Translateforthecraft/drf-auth-kit | 121 | — | ~2.4k | Automated safety check: Notes | MIT | |
| Docs Leadlablup/backend.ai-webui | 133 | — | ~2.9k | Automated safety check: Pass | LGPL-3.0 | |
| Canvas HumanizerX-isdoingreat/canvas-pilot | 125 | — | ~10k | Automated safety check: Notes | AGPL-3.0 | |
| Yao Meta Skillyaojingang/yao-meta-skill | 2.7k | — | ~768 | Automated safety check: Pass | MIT |
Similar skills
Translate
forthecraft/drf-auth-kit
Run Django translation workflow - extract untranslated strings, translate them to 59 languages using parallel translator subagents, and apply back to PO files.
Docs Lead
lablup/backend.ai-webui
A skill your agent uses whenever the user mentions docs, the manual, documentation, terminology, translations, or screenshots — including indirect mentions like "이 PR 문서 영향 봐줘", "문서 점검", "용어 통일"…
Canvas Humanizer
X-isdoingreat/canvas-pilot
Reduces AI-detection signals in drafted text by routing every non-locked sentence through round-trip translation (English → intermediate language → English) and selecting the candidate that…
Yao Meta Skill
yaojingang/yao-meta-skill
Create, improve, or evaluate an existing skill from workflows, prompts, SOPs, scripts.
Translate Book
deusyu/translate-book
Translate books (PDF/DOCX/EPUB) into any language using parallel sub-agents.
More from shapeshift/web
React Best Practices
shapeshift/web
Comprehensive React and Next.js performance optimization guide with 40+ rules for eliminating waterfalls, optimizing bundles, and improving rendering.
Qabot Fixture
shapeshift/web
Create a new qabot E2E test fixture interactively. An agent skill from shapeshift/web.
Translate
shapeshift/web
Translate new/changed English UI strings into all supported languages using a translate-review-refine pipeline.
Chain Integration
shapeshift/web
Integrate a new blockchain as a second-class citizen in ShapeShift Web.
Qabot
shapeshift/web
Run QA tests using agent-browser and post results to the qabot dashboard.
Swapper Integration
shapeshift/web
Integrate new DEX aggregators, swappers, or bridge protocols (like Bebop, Portals, Jupiter, 0x, 1inch, etc.) into ShapeShift Web.
Related
Categories
Questions about Benchmark Translate
What does Benchmark Translate do?
Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report. Benchmark Translate is an agent skill from shapeshift/web. Run a quality benchmark of the /translate skill by selecting stratified test keys, capturing ground truth, translating, judging with sub-agents, and compiling a regression report.
When should I use Benchmark Translate?
Benchmark Translate fits situations like: tasks that involve Translation; tasks that involve Subagents; tasks that involve LLM evaluation.
How do I install Benchmark Translate in Claude Code?
Run `npx skills add shapeshift/web --skill benchmark-translate -a claude-code`. Or copy the skill folder (.claude/skills/benchmark-translate in shapeshift/web) into .claude/skills/benchmark-translate in your project. Claude Code loads it when a task matches its description.
How do I install Benchmark Translate in Codex?
Run `npx skills add shapeshift/web --skill benchmark-translate -a codex`. Or copy the skill folder (.claude/skills/benchmark-translate in shapeshift/web) into .agents/skills/benchmark-translate in your project. Codex loads it when a task matches its description.
Can I use Benchmark Translate in Cursor, Gemini CLI or GitHub Copilot?
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add shapeshift/web --skill benchmark-translate -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/benchmark-translate, .gemini/skills/benchmark-translate, .github/skills/benchmark-translate and .opencode/skills/benchmark-translate in your project.
What does Benchmark Translate need to run?
Going by SKILL.md and its folder, Benchmark Translate needs JavaScript for the scripts in its folder and the command-line tools its instructions call (node and git). Our summary lists: Node.js. Its frontmatter pre-approves these tools: Read, Write, Edit, Grep, Glob, Bash(node *), Bash(git checkout*), Bash(git diff*), Bash(git status*), Bash(git rev-parse*), Task, Skill, AskUserQuestion.
Does Benchmark Translate access the network?
SKILL.md contains no URLs. Its commands use git, which can reach the network depending on how they are called. This is read from the text; nothing was executed.
Is Benchmark Translate safe to install?
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.
What licence does Benchmark Translate use?
Benchmark Translate is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
How many tokens does Benchmark Translate use?
About 1.6k tokens (SKILL.md is roughly 6.2k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
What are the alternatives to Benchmark Translate?
Skills that share tags, products or a category with Benchmark Translate: Po Translate (natsukium/dotfiles, 106 stars), Translate (forthecraft/drf-auth-kit, 121 stars), Docs Lead (lablup/backend.ai-webui, 133 stars) and Canvas Humanizer (X-isdoingreat/canvas-pilot, 125 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
Who maintains Benchmark Translate?
shapeshift (a GitHub organization) maintains it in shapeshift/web, which has 206 GitHub stars. The repository holds 7 skills in this directory. The repository was last updated on October 7, 2026.
Source: shapeshift/web on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.