Hallucinating Labels

hallucinating-labels

Ask a cheap model to write a plausible label for the item. Snap that label onto the real vocabulary with an embedder. The model never sees the label set.

Doug Turnbull's pattern (softwaredoug.com, 2026-08-10), with the two prompt and boundary corrections that measurement produced.

Check the boundary first

If the whole vocabulary fits in a prompt, do not use this skill. Ship the label list and ask for a constrained choice. Measured on WANDS (860 labels, 468 queries, one gold label each, gemini-3.5-flash-lite):

Shipping the vocabulary is 14 points more accurate and 880× more expensive. Take the accuracy unless the tokens are the problem. The tokens are the problem when the vocabulary does not fit, when a provider enum cap rejects it, or when per-call cost at volume dominates — a 5,000-label vocabulary is roughly 30k tokens on every single call.

This skill still beats every model-free baseline by a wide margin, so it is the right tool whenever shipping the vocabulary is off the table.

Procedure

1. Write the vocabulary to a file, one label per line, and index it once.

python3 scripts/snap.py build --vocab categories.txt --out .snap-index.pkl

Default backend is tfidf — sklearn only, no download. Pass --backend minilm when sentence-transformers and a ~90 MB download are available and the items share no wording with the labels; it scored 0.564 to tfidf's 0.528 on WANDS. Where items literally contain their own label words, tfidf wins outright (0.416 vs 0.356 on a memory-tag corpus).

2. Write the labels yourself, in batches of 40, using the register prompt below. Write them to a file, one per line, in the same order as the items.

3. Snap.

python3 scripts/snap.py snap --index .snap-index.pkl --labels written.txt --k 3

Add --min-score 0.35 to get null instead of a bad snap, and --items items.txt --union for long items (see below). Output is JSON with the top-k legal labels per item.

4. Report the nulls and the low scores. A snap at cosine 0.18 is noise wearing a legal label. Never present one as a classification.

Anchor the prompt on REGISTER, not on novelty

This is the correction that matters most, and it is the opposite of what the source post's prompt says. Its prompt opens "create a novel, never-seen-before classification". That instruction is safe only with a model too weak to follow it.

Measured on the same 40 WANDS queries, MiniLM backend:

Haiku obeyed. Asked for novelty it produced novelty — Hydraulic Styling Thrones, Weathered Branch-Frame Reflectors, Chromatic Comfort Accents — and scored a fifth of what doing nothing scores. Gemini flash-lite half-ignored the same instruction and wrote Salon & Styling Chairs, Rustic Wall Mirrors, which is what the snap needs. The pattern wants a novel instance in the vocabulary's register, and "never-seen-before" asks for novel wording. A better instruction-follower is worse at the badly-worded prompt.

The register prompt also beat the novelty prompt on Gemini across all 468 queries (0.564 vs 0.489 acc@1), so it is strictly the better wording. Use this:

You are writing entries for a {DOMAIN} vocabulary.

For each item below, write the label that this vocabulary WOULD file that item under. Write it the way the vocabulary writes labels — match the examples' register, length and wording exactly.

Do not worry about whether the label already exists. Write the obvious one. Do not invent novel or creative wording, do not use marketing adjectives, do not hedge, do not explain.

Examples of the register: {6-8 REAL LABELS FROM THE VOCABULARY}

Output one line per item, in the same order, formatted exactly as: <n>. <label>

ITEMS: {NUMBERED ITEMS}

The examples are load-bearing — they are how the register gets communicated. Draw 6-8 real labels from the vocabulary. They are not the vocabulary; sending eight labels is not sending five thousand.

Batch 40 per call. Never one item per call

Batching is free: 0.496/0.641 batched ×40 against 0.489/0.613 unbatched, at 1/17 the input tokens and 1/9 the wall-clock.

Write the labels yourself when the items are already in context — you are the cheap model here, and it costs one short generation. Delegate to a Haiku subagent only in batches, and only when the item list is long enough to be worth it. A subagent invocation carries a measured floor of ~32,500 tokens before it reads your prompt: a general-purpose Haiku subagent asked to output the single word ok, with zero tool calls, spent 32,539. Per item that floor is 813 tokens at batch 40 and 32,500 at batch 1.

Parse the model's numbered reply back by index, not by zipping positionally. When the model drops item 2 of 40, zipping shifts every later item onto its neighbour's label and nothing signals it. A dropped item is an empty label and then a null.