Embeddings Search

embeddings-search — make and judge the vectors

You own the embedding technique layer: turn a corpus into searchable vectors, turn a question into a good retrieval, and measure whether that retrieval is any good. You stop the moment the right chunks come back, measured by a number. You do not assemble a prompt or generate an answer.

Route the adjacent surfaces away:

• Operating the store — collection schema, HNSW/IVFFlat tuning, metadata-filter path, quantization, ef_search recall knobs → ../vector-db/SKILL.md. You decide what vectors go in and how to query; vector-db decides how the store holds and serves them.
• The full retrieve → rerank → prompt → generate → answer loop and its groundedness / faithfulness eval → ../rag/SKILL.md.
• Pulling typed fields out of documents (invoice number, date, total) → ../structured-extraction/SKILL.md.
• Writing the prompt the model reasons with → ../prompt-engineering/SKILL.md.

1. Pick the embedding model

Decide on three axes: language coverage, quality tier (read MTEB but don't worship it), and cost — where cost is set by dimensions, because dims set storage and memory.

Quality anchor (mid-2026 MTEB English retrieval): Gemini ~68.3, Cohere embed-v4 ~65.2, OpenAI 3-large ~64.6, BGE-M3 ~63.0. MTEB is the standard comparison, not a verdict on your domain — references/models.md carries the full model matrix (dims, max tokens, price, input_type convention, Matryoshka support) and how to read MTEB without over-trusting it.

Two hard rules — each is a silent failure, no error, just worse results:

• Match the distance metric to the model. A cosine-trained model indexed or queried with L2 ranks silently wrong. Cosine → <=> in pgvector / Distance.COSINE in Qdrant. The index operator itself is vector-db's job; the requirement originates from the model, so state it in your config.
• Respect query/document asymmetry. Cohere, Voyage, Gemini, e5, BGE expect a different prompt or input_type for the query vs the stored passage. Embed both sides identically and recall silently drops.

Dimensions = cost. A 1024-dim float32 vector is 4 KB; at 10M docs that is 40 GB, and doubling dims doubles storage and memory. Matryoshka-trained models (OpenAI 3-*, Cohere, Gemini) let you truncate dims for graceful degradation — never re-embed the whole corpus just to shrink vectors.

2. Chunk the corpus

Start boring. Upgrade only when a number tells you to.

# Default recipe: recursive split, TOKEN-accurate count, 10–20% overlap.
import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

enc = tiktoken.get_encoding("cl100k_base")
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,      # tokens, not characters
    chunk_overlap=64,    # ~12% — keeps sentences from being cut mid-thought
)
chunks = splitter.split_text(document_text)

Counting by characters instead of tokens is the most common own-goal: 512 characters is ~100–130 tokens of English and far fewer of CJK, so "512" silently means different things per language and per model limit.

Upgrade ladder — graduate only when retrieval metrics (section 6) justify the added compute:

Embed the searchable text; store the rest as metadata. What you embed is what gets matched — don't bury the answer text under boilerplate, and don't embed raw HTML.

3. Construct the query

The query is half the retrieval. Embed it the way the model expects, then improve it only when recall data says you should.

# Asymmetric model: query and document use DIFFERENT input_type. Getting this wrong is silent.
q_vec   = embed(text=user_question, input_type="search_query")     # Cohere / Voyage
d_vec   = embed(text=passage,       input_type="search_document")
# e5 / BGE convention is a textual prefix instead:
#   query    -> "query: how do refunds work"
#   passage  -> "passage: Refunds are processed within 14 days…"

Query-side techniques, when each pays:

• Query rewriting — when user queries are terse or full of pronouns; normalise before embedding.
• HyDE (embed a hypothetical answer, not the question) — when questions are short and answers are long/technical, so the answer-shaped vector lands nearer the passage.
• Multi-query (fan out 3–4 paraphrases, union the hits) — when one phrasing under-recalls; costs N embeds and a dedup.