NLP

nlp — tokenize the text, pick the model type, pick the metric

You own the language-modeling discipline: how raw text becomes tokens, which transformer architecture fits a task, and which metric actually tells you whether it worked. When the question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3× the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the RAG loop, prompt wording, and the training step itself — those route out (below).

Route out first (loud — do not duplicate these)

• Retrieval embeddings + vector search (which embedding model, chunking, recall@k, rerank) → ../embeddings-search/SKILL.md. Sentence embeddings live here as a task; using them to retrieve is theirs.
• The retrieve → prompt → generate → answer loop and groundedness → ../rag/SKILL.md.
• Prompt wording / few-shot / system prompts → ../prompt-engineering/SKILL.md.
• Training the network (LoRA/SFT, trainer loop, PyTorch) → ../finetuning/SKILL.md
  • ../deep-learning/SKILL.md.
• The training corpus itself (JSONL messages, label sets) → ../training-data/SKILL.md.

Decision: model type per task (get this right before anything else)

Pick the architecture from the task's shape, not from what is trendy. A decoder LLM can technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more accurate on a fixed-label task.

* Architecture families are stable; specific checkpoints and their licenses are not — check the HF model card before you commit (licenses change; see open-weights). ModernBERT (2024) is a current long-context encoder; verify the latest at author time.

The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an encoder does it for a fraction of the cost), and forcing an encoder to generate (it cannot — it has no decoder).

1. Tokenization

Every downstream number depends on this step, and its failures are silent. The single load-bearing rule:

Load the tokenizer that shipped with the checkpoint, and use the same one at train and inference. AutoTokenizer.from_pretrained(same_checkpoint). A train/inference tokenizer mismatch — different vocab, different special tokens, different casing/normalization — maps text to token ids the model never saw and corrupts everything downstream with no error.

from transformers import AutoTokenizer   # transformers current major ~v5 (verify at author time)

tok = AutoTokenizer.from_pretrained("bert-base-cased")
enc = tok("Tokenizers matter.", return_offsets_mapping=True)
tok.convert_ids_to_tokens(enc["input_ids"])
# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]']  — note WordPiece '##' continuation + added specials

The four algorithms (full mechanics in references/tokenization.md):

• Byte-level BPE has no [UNK]. Base vocab is exactly 256 (all byte values), so every emoji, accent, and script maps to some byte sequence — nothing falls out as unknown (verified: HF NLP course ch.6). WordPiece/word-level tokenizers do have [UNK] and lose OOV content.
• SentencePiece is reversible — it treats space as a normal symbol (the ▁ meta-symbol), so decode(encode(x)) == x without language-specific detokenization rules. That is why it dominates multilingual models.

Special tokens are not decoration. [CLS]/<s> carries the pooled sentence representation for classification; [SEP]/</s> marks segment/end; [PAD] fills a batch (and must be masked out via attention_mask); [MASK] is the MLM target; [UNK] is the fallback. Names differ by model ([CLS] in BERT vs <s> in RoBERTa) — another reason to never hand-roll the tokenizer.

Why it matters — three concrete costs:

• $ cost & context. Token count is the bill and the context budget. Fewer tokens per sentence = cheaper calls and more room in the window.
• OOV / information loss. A tokenizer that emits [UNK] throws away content it can't represent; byte-level/SentencePiece degrade gracefully instead.
• Fairness. Vocab trained mostly on English fragments other scripts far harder — the same meaning costs more tokens, more money, and more latency (section 5).

2. Tasks

Text classification (encoder + classification head)

from transformers import pipeline
clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
clf("The service was slow but the food was incredible.")
# [{'label': 'POSITIVE', 'score': 0.99...}]

Metric: accuracy on balanced data; macro-F1 the moment classes are imbalanced (accuracy lies when 95% of rows are one class).

Token classification / NER (encoder, per-token labels)

Labels are B-/I-/O spans aligned to subword tokens: the first subword of a word gets the label, continuation subwords and special tokens get -100 (ignored by the loss). Evaluate with seqeval at the entity level, never per-token accuracy (per-token accuracy is inflated by the flood of O tokens).

from transformers import pipeline
ner = pipeline("token-classification", aggregation_strategy="simple")
ner("Ada Lovelace worked in London.")
# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'

Seq2seq — summarization / translation (encoder-decoder)

summ = pipeline("summarization", model="facebook/bart-large-cnn")
summ(long_article, max_length=130, min_length=30)

Metric: ROUGE for summarization, BLEU/chrF for translation — with the heavy caveat in section 4.

Sentence embeddings (SBERT — the task, not retrieval)

from sentence_transformers import SentenceTransformer   # sentence-transformers ~v5 (verify)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
model.similarity(emb, emb)   # semantic textual similarity / clustering / paraphrase mining

Producing/judging embeddings for retrieval (model choice, chunking, recall@k, rerankers) is embeddings-search, not here.