Agent skill

NLP Preprocessing Toolkit

by revfactory in revfactory/harness-100

Text preprocessing technique catalog: tokenization, normalization, stopwords, morphological analysis, embedding selection, and language-specific processing guides.

Apache-2.0Auto-check passedAI & LLM Engineering

Install NLP Preprocessing Toolkit

skills CLI
$ npx skills add revfactory/harness-100 --skill nlp-preprocessing-toolkit -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install revfactory/harness-100 nlp-preprocessing-toolkit --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/revfactory/harness-100.git skills-src && mkdir -p .claude/skills && cp -r skills-src/en/33-text-processor/.claude/skills/nlp-preprocessing-toolkit .claude/skills/nlp-preprocessing-toolkit && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
nlp-preprocessing-toolkit
GitHub stars
1.3k
Token cost
~1.4k tokens
SKILL.md length
214 words
Files
1
Skills in repo
464
Repo updated
First seen
Licence
Apache-2.0

At a glance

Text preprocessing technique catalog: tokenization, normalization, stopwords, morphological analysis, embedding selection, and language-specific processing guides.

  • Requests involving text preprocessing
  • SKILL.md covers Preprocessing Pipeline, Language-Specific Processing…, Text Vectorization and Text Quality Metrics, plus 1 more section
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md
  • Morphological analysis

What it does

NLP Preprocessing Toolkit is an agent skill from revfactory/harness-100. Text preprocessing technique catalog: tokenization, normalization, stopwords, morphological analysis, embedding selection, and language-specific processing guides. Use this skill for requests involving 'text preprocessing', 'tokenization', 'morphological analysis', 'KoNLPy', 'stopwords', 'normalization', 'TF-IDF', 'embeddings', 'Word2Vec', 'NLP preprocessing', etc. Enhances the text processing capabilities of the preprocessor and extractor agents. Note: sentiment analysis models and classification algorithm…

Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering, covering Natural language processing, Embeddings and Database schema design. It works with Java. The licence is Apache-2.0.

When your agent uses it

  • Requests involving text preprocessing
  • Morphological analysis
  • NLP preprocessing

Example prompts

  • “text preprocessing”
  • “tokenization”
  • “morphological analysis”
  • “/nlp-preprocessing-toolkit”

Requirements

  • Python 3

What it can do on your machine

Read from SKILL.md and the folder at commit 8e8d35c. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

NLP Preprocessing Toolkit loads about 1.4k tokens when it runs. Until then it costs about 146 tokens; SKILL.md has 214 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~146
When it runs · the whole SKILL.md, loaded when a task matches
~1.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from revfactory/harness-100 at commit 8e8d35c, republished under its Apache-2.0 licence (© revfactory). 214 words, ~1,372 tokens.

Download SKILL.mdSave it as .claude/skills/nlp-preprocessing-toolkit/SKILL.md (or your agent's skills folder).
name
nlp-preprocessing-toolkit
description
Text preprocessing technique catalog: tokenization, normalization, stopwords, morphological analysis, embedding selection, and language-specific processing guides. Use this skill for requests involving 'text preprocessing', 'tokenization', 'morphological analysis', 'KoNLPy', 'stopwords', 'normalization', 'TF-IDF', 'embeddings', 'Word2Vec', 'NLP preprocessing', etc. Enhances the text processing capabilities of the preprocessor and extractor agents. Note: sentiment analysis models and classification algorithm selection are outside the scope of this skill.

NLP Preprocessing Toolkit — Text Preprocessing Tools Guide

A catalog of preprocessing techniques for transforming text data into analysis-ready formats.

Preprocessing Pipeline

Raw Text
├── 1. Encoding normalization (UTF-8)
├── 2. HTML/special character removal
├── 3. Unicode normalization (NFKC)
├── 4. Lowercasing (for alphabetic scripts)
├── 5. Tokenization
├── 6. Stopword removal
├── 7. Morphological analysis / stemming
├── 8. Regex filtering
└── 9. Vectorization (TF-IDF / embeddings)

Language-Specific Processing (Korean)

Morphological Analyzer Comparison
AnalyzerSpeedAccuracyCustom DictionaryInstallation
MecabFastestHighYesC dependency
Okt (Twitter)FastMediumYesJava dependency
KomoranMediumHighYesJava dependency
KkmaSlowHighNoJava dependency
KiwiFastHighYesPython native
python
# Kiwi (easiest installation, excellent performance)
from kiwipiepy import Kiwi
kiwi = Kiwi()

tokens = kiwi.tokenize("FatherEnteredTheRoom")
# [Token(form='Father', tag='NNG'), Token(form='subject', tag='JKS'),
#  Token(form='Room', tag='NNG'), Token(form='to', tag='JKB'),
#  Token(form='entered', tag='VV'), Token(form='hon', tag='EP'),
#  Token(form='past', tag='EP'), Token(form='decl', tag='EF')]

# Extract nouns only
nouns = [t.form for t in tokens if t.tag.startswith('NN')]
Korean Text Normalization
python
import re, unicodedata

def normalize_korean(text):
    # Unicode normalization (compatibility decomposition + canonical composition)
    text = unicodedata.normalize('NFKC', text)

    # Remove repeated characters ("xxxxx" -> "xx")
    text = re.sub(r'(.)\1{2,}', r'\1\1', text)

    # Remove standalone consonants/vowels (repeated consonants/vowels, etc. may be preserved for sentiment analysis)
    # text = re.sub(r'[\u3131-\u3163]+', '', text)

    # Keep only alphanumeric characters, Korean characters, and whitespace
    text = re.sub(r'[^\w\s\u3131-\uD79D]', ' ', text)

    # Remove multiple whitespace
    text = re.sub(r'\s+', ' ', text).strip()

    return text
Korean Stopwords
python
KOREAN_STOPWORDS = {
    # Particles (Korean grammatical markers)
    'i', 'ga', 'eun', 'neun', 'eul', 'reul', 'e', 'ui', 'wa', 'gwa',
    'do', 'ro', 'eseo', 'kkaji', 'buteo', 'man', 'euro',
    # Pronouns
    'geu', 'i', 'jeo', 'geot', 'su', 'deung', 'deul',
    # Adverbs
    'maeu', 'aju', 'jeongmal', 'neomu', 'jal', 'tto', 'deo',
    # Conjunctions/Interjections
    'geurigo', 'hajiman', 'geureonde', 'geuraeseo',
}

Text Vectorization

TF-IDF
python
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(
    max_features=10000,
    min_df=2,           # Must appear in at least 2 documents
    max_df=0.95,        # Exclude if appearing in more than 95% of documents
    ngram_range=(1, 2), # Unigrams + bigrams
    sublinear_tf=True,  # 1 + log(tf) — dampens high-frequency terms
)
tfidf_matrix = vectorizer.fit_transform(texts)
Embedding Selection Guide
MethodDimensionsBest ForCharacteristics
TF-IDFHigh-dimensional (sparse)Keyword-centric, small-scaleInterpretable, fast
Word2Vec100-300Similarity, analogiesWord-level, limited context
FastText100-300Korean, OOV handlingSubword-based, robust to unseen words
BERT768Classification, NER, QAContext-dependent, bidirectional
Sentence-BERT384-768Document similarity, searchSentence-level embeddings
python
# Sentence-BERT (Korean)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer('jhgan/ko-sroberta-multitask')
embeddings = model.encode(texts, show_progress_bar=True)
# Cosine similarity
from sklearn.metrics.pairwise import cosine_similarity
sim_matrix = cosine_similarity(embeddings)

Text Quality Metrics

MetricCalculationThreshold
Average token countTokens per text< 3 indicates analysis limitations
Vocabulary diversityUnique tokens / total tokens0.2-0.8 is acceptable
Language purityProportion of primary language> 90% recommended
Missing rateProportion of empty texts< 5%
Duplication rateProportion of identical texts< 10%

Preprocessing Decision Checklist

  • Encoding issues resolved (e.g., CP949)
  • HTML tags/URLs removed
  • Emoji handling decided (remove vs. convert to text vs. use for sentiment)
  • Number handling decided (remove vs. tokenize vs. replace with [NUM])
  • Morphological analyzer selected
  • Stopword list customized for domain
  • Minimum token count filtering applied
  • Vectorization method selected

© revfactory, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in en/33-text-processor/.claude/skills/nlp-preprocessing-toolkit of revfactory/harness-100.

Open the folder on GitHubat commit 8e8d35c

Compare with similar skills

NLP Preprocessing Toolkit next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

NLP Preprocessing Toolkit compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
NLP Preprocessing Toolkit this skillrevfactory/harness-1001.3k—~1.4kAutomated safety check: PassApache-2.0
Gptqmodel Tokenizer NormalizationModelCloud/GPTQModel1.3k—~1.1kAutomated safety check: PassCustom licence
Comparetaishi-i/awesome-japanese-nlp-resources1k—~4.1kAutomated safety check: NotesCC0-1.0
Sentence Transformers EmbeddingsOrchestra-Research/AI-Research-SKILLs13k3 repos~1.6kAutomated safety check: PassMIT
Researchtaishi-i/awesome-japanese-nlp-resources1k—~3.5kAutomated safety check: NotesCC0-1.0
Searchtaishi-i/awesome-japanese-nlp-resources1k—~4.3kAutomated safety check: NotesCC0-1.0

Similar skills

  • Diagnose and correct GPT-QModel tokenizer initialization, tokenization normalization, special-token handling, prompt rendering, and chat-template problems.

    1.3k GitHub stars~1.1k tokensUpdated today
    AI & LLM EngineeringAuto-check passed
  • Compare

    taishi-i/awesome-japanese-nlp-resources

    Compare several Japanese NLP libraries, models, or datasets for a keyword (a specific tool name, or a function/task like '形態素解析') across a handful of criteria chosen for that comparison, rendered as…

    1k GitHub stars~4.1k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check: notes
  • Sentence Transformers Embeddings

    Orchestra-Research/AI-Research-SKILLs

    Generates text embeddings locally with the sentence-transformers library for RAG, semantic search, clustering and similarity, with model picks for general, multilingual and legal text.

    13k GitHub starsUsed in 3 repos~1.6k tokens
    AI & LLM EngineeringAuto-check passed
  • Research

    taishi-i/awesome-japanese-nlp-resources

    Analyze current trends and challenges in Japanese NLP for a topic.

    1k GitHub stars~3.5k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check: notes
  • Search

    taishi-i/awesome-japanese-nlp-resources

    Search all Japanese NLP resources (libraries, models, datasets, tutorials, dictionaries, Hugging Face).

    1k GitHub stars~4.3k tokensUpdated yesterday
    AI & LLM EngineeringAuto-check: notes
  • Langchain4j RAG Implementation Patterns

    giuseppe-trisciuoglio/developer-kit

    Provides Retrieval-Augmented Generation (RAG) implementation patterns with LangChain4j for Java.

    355 GitHub starsUsed in 1 repo~3.3k tokens
    AI & LLM EngineeringAuto-check: notes

More from revfactory/harness-100

All 464 skills in this repo
  • Anti Bot Analyzer

    revfactory/harness-100

    A skill for analyzing website anti-bot defense mechanisms and developing legitimate evasion strategies.

    1.3k GitHub stars~1.1k tokensUpdated 6 mo ago
    Auto-check passed
  • API Error Design Patterns

    revfactory/harness-100

    Reference for designing how an API reports failures: structured error codes, response shapes, client-friendly messages, an error catalog and retry or fallback advice.

    1.3k GitHub stars~1.6k tokensUpdated 6 mo ago
    Auto-check passed
  • API Security Checklist

    revfactory/harness-100

    Walks a backend-dev agent through OWASP API Top 10 checks, authentication and authorization patterns, and defense code during API design.

    1.3k GitHub stars~1.7k tokensUpdated 6 mo ago
    Auto-check passed
  • Arg Parser Generator

    revfactory/harness-100

    Methodology for systematically designing and generating CLI tool argument parser structures.

    1.3k GitHub stars~1.2k tokensUpdated 6 mo ago
    Auto-check passed
  • Audience Segmentation

    revfactory/harness-100

    Audience segmentation skill used by the analyst and curator agents.

    1.3k GitHub stars~1.3k tokensUpdated 6 mo ago
    Auto-check passed
  • Audio Storytelling

    revfactory/harness-100

    Audio storytelling skill used by the podcast scriptwriter and show note editor.

    1.3k GitHub stars~1.6k tokensUpdated 6 mo ago
    Auto-check passed

Works with

Questions about NLP Preprocessing Toolkit

What does NLP Preprocessing Toolkit do?

Text preprocessing technique catalog: tokenization, normalization, stopwords, morphological analysis, embedding selection, and language-specific processing guides. NLP Preprocessing Toolkit is an agent skill from revfactory/harness-100. Text preprocessing technique catalog: tokenization, normalization, stopwords, morphological analysis, embedding selection, and language-specific processing guides.

When should I use NLP Preprocessing Toolkit?

NLP Preprocessing Toolkit fits situations like: requests involving text preprocessing; morphological analysis; NLP preprocessing.

How do I install NLP Preprocessing Toolkit in Claude Code?

Run `npx skills add revfactory/harness-100 --skill nlp-preprocessing-toolkit -a claude-code`. Or copy the skill folder (en/33-text-processor/.claude/skills/nlp-preprocessing-toolkit in revfactory/harness-100) into .claude/skills/nlp-preprocessing-toolkit in your project. Claude Code loads it when a task matches its description.

How do I install NLP Preprocessing Toolkit in Codex?

Run `npx skills add revfactory/harness-100 --skill nlp-preprocessing-toolkit -a codex`. Or copy the skill folder (en/33-text-processor/.claude/skills/nlp-preprocessing-toolkit in revfactory/harness-100) into .agents/skills/nlp-preprocessing-toolkit in your project. Codex loads it when a task matches its description.

Can I use NLP Preprocessing Toolkit in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add revfactory/harness-100 --skill nlp-preprocessing-toolkit -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/nlp-preprocessing-toolkit, .gemini/skills/nlp-preprocessing-toolkit, .github/skills/nlp-preprocessing-toolkit and .opencode/skills/nlp-preprocessing-toolkit in your project.

What does NLP Preprocessing Toolkit need to run?

SKILL.md names no scripts, command-line tools or credentials: NLP Preprocessing Toolkit is instructions for the agent only. Our summary lists: Python 3.

Does NLP Preprocessing Toolkit access the network?

SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.

Is NLP Preprocessing Toolkit safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does NLP Preprocessing Toolkit use?

NLP Preprocessing Toolkit is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does NLP Preprocessing Toolkit use?

About 1.4k tokens (SKILL.md is roughly 5.5k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to NLP Preprocessing Toolkit?

Skills that share tags, products or a category with NLP Preprocessing Toolkit: Gptqmodel Tokenizer Normalization (ModelCloud/GPTQModel, 1.3k stars), Compare (taishi-i/awesome-japanese-nlp-resources, 1k stars), Sentence Transformers Embeddings (Orchestra-Research/AI-Research-SKILLs, 13k stars) and Research (taishi-i/awesome-japanese-nlp-resources, 1k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains NLP Preprocessing Toolkit?

revfactory (a GitHub user) maintains it in revfactory/harness-100, which has 1,290 GitHub stars. The repository holds 464 skills in this directory. The repository was last updated on March 22, 2026.

Source: revfactory/harness-100 on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.