Agentsop Multi Tenant RAG

Multi-Tenant RAG · Security-First Isolation SOP

Third-person operating model for a coder agent that owns retrieval correctness across tenant boundaries. The audience is the LLM agent writing or reviewing the code — not the end user.

One sentence: Isolation lives at the vector store query boundary, not at the model. Anything that reaches the LLM's context window has already leaked.

1. 何时激活 (Activation Rules)

Activate this skill whenever any of the following holds:

1. The codebase contains a retrieval call (vector_store.query, query_points, similarity_search, as_retriever().retrieve(...), raw pgvector ORDER BY embedding <-> $1) and the corpus serves more than one tenant, customer, organisation, workspace, user, or permission scope.
2. The user mentions any of: multi-tenant RAG, namespace, tenant, workspace, tenant_id, org_id, user_id, "cross-customer", "shared index", "knowledge base per team".
3. A bug report says "User A saw User B's document", "wrong company's data surfaced", "the assistant cited a doc I don't have access to", or anything that smells like cross-context bleed.
4. PR review: any new code calling a vector store without a tenant-scoped filter argument, or filtering only on the returned nodes / documents list after retrieval.
5. A new RAG endpoint is about to ship and tenant scoping has not been explicitly audited.

Do not activate when:

• The corpus is fully public and there is no per-tenant view (e.g. open-data Q&A).
• The retrieval pipeline already enforces a physically separate index / collection per tenant via infrastructure the application code cannot override (e.g. one Pinecone index per customer with credentials issued per-tenant). In that case the isolation lives in IAM, not in this skill.

2. 核心心智模型 (Core Mental Model)

Three principles. If a design violates any of them the system is exploitable, regardless of how good the LLM prompt is.

Principle 1 — Isolation lives at the boundary, not at the model

The boundary is the vector store query call. Whatever crosses that boundary is in the trust set. An LLM, a reranker, or a postprocessor that "filters out" foreign-tenant chunks is operating inside the breach: those tokens have already been embedded, retrieved, scored, and exposed to attacker-controlled prompts. CVE-2024-41892 (Pinecone, 2024) is the canonical demonstration — RBAC checks executed after retrieval allowed sentinel data to cross namespace boundaries before the access check fired. (See we45, CSO Online references.)

Operational corollary: any code shaped results = vs.query(...); results = [r for r in results if r.metadata["tenant"] == ctx.tenant] is a security defect, not a style nit. The leak already happened — you only hid it from the user.

Principle 2 — Embed the tenant key at ingestion, not just at query

A query-time filter is only enforceable if every vector carries the key. The two failure shapes:

• Forgotten field: a single document ingested without tenant_id becomes a shared global record returned to all tenants — it matches no filter predicate that requires equality.
• Mutable field: tenant_id derived from the document body / LLM extraction rather than from the request's authenticated session — attacker can craft document content that re-labels itself.

The tenant key must come from the authenticated session at ingestion time and be written into the immutable payload / metadata column. Treat it like a foreign key to your tenants table.

Principle 3 — Defence in depth: namespace and filter, not namespace or filter

Most production-grade stacks layer two mechanisms:

Pinecone's official guidance is "namespace per tenant in serverless"; Weaviate mandates a tenant handle on every CRUD when multiTenancyConfig.enabled=true; Qdrant marks the field as is_tenant=true and requires Filter.must on read; pgvector pairs schema partitioning with RLS. The principle is the same: one layer is one bug away from failing open.

3. SOP 工作流 (Agentic Protocol)

Eight stages. Each gates the next. Stop and remediate at the first failure.

Stage 0 — Threat-model the boundary

Before any code change, write down:

1. Tenant key: what identifier separates customers? (tenant_id, org_id, workspace_id, user_id, acl_label). Pick one — do not mix.
2. Trust source: where does the request-time tenant key come from? It must be from the authenticated session / JWT claim / signed header — never from a query param, prompt, or document body.
3. Cardinality: how many tenants? (<100 → namespace/collection viable;

   10k → payload partition + index is mandatory). Drives Stage 2 choice.

4. Blast radius: if a single vector leaks, what's the worst document? (PII / regulated / IP / public). Raises bar on Stages 4-6.

Artifact: a 5-line THREAT_MODEL.md snippet co-located with the retriever.

Stage 1 — Pick the isolation primitive (vendor-specific)

Decision rule: prefer the strongest physical primitive your tenant cardinality can afford, then always add the filter at query time anyway (Principle 3).

Stage 2 — Ingestion: write the key, lock the schema

# Canonical shape — vendor-agnostic
node.metadata = {
    "tenant_id": session.tenant_id,        # MUST come from authenticated session
    "doc_id":    doc.id,
    "source":    doc.uri,
    "ingested_at": now_utc(),
    # ... domain fields
}
assert node.metadata["tenant_id"], "refuse to embed without tenant_id"

Hard rules:

• Refuse to write a vector if tenant_id is missing or empty — fail loud, do not default.
• tenant_id is never taken from document content; never mutable post-ingestion.
• Schema-validate metadata before embedding (Pydantic / JSON Schema) — type drift breaks $eq silently.
• For Qdrant: create the payload index before the first upsert (create_payload_index(field="tenant_id", schema="keyword", is_tenant=True)).
• For Weaviate: create the tenant (collection.tenants.create([Tenant(name=t)])) before any write.

Stage 3 — Query-time filter (the only place the rule applies)

Every retrieval call must accept a tenant key from the session and pass it into the vendor's filter argument. Templates in OP-04 through OP-08.

# Generic LlamaIndex shape — works across Pinecone, Qdrant, Weaviate, Chroma
from llama_index.core.vector_stores import (
    MetadataFilter, MetadataFilters, FilterOperator,
)

filters = MetadataFilters(filters=[
    MetadataFilter(key="tenant_id", value=session.tenant_id,
                   operator=FilterOperator.EQ),
])
retriever = index.as_retriever(similarity_top_k=8, filters=filters)

If the framework or vendor SDK does not expose a query-time filter, switch the framework or vendor — do not patch with a post-filter.

Stage 4 — Cross-tenant property test (the gate)

Before merging, the PR must include — and CI must run — a test that:

1. Ingests N≥3 documents per tenant for at least 2 tenants (A and B).
2. Issues a query as tenant A whose semantic top-1 in the unfiltered search would be a tenant-B document (craft the query against B's content).
3. Asserts zero tenant-B results returned.
4. Repeats symmetrically as tenant B.

If the unfiltered top-1 is not a foreign-tenant doc, the test is dishonest — rewrite the corpus to make foreign-tenant semantically closer, otherwise the test always passes vacuously.

See OP-09 for a runnable scaffold.

Stage 5 — Defense in depth (Principle 3)

Layer at least one of:

• Per-tenant namespace / shard / collection on top of the metadata filter.
• Per-tenant API key / scoped credential at the vector store layer (Pinecone API key project scope, Postgres role per tenant).
• DB-level enforcement (pgvector RLS policy USING (tenant_id = current_setting('app.current_tenant')::uuid)).

A single layer is one config bug away from open.

Stage 6 — Audit logging

Every retrieval call logs:

{ts, request_id, session.tenant_id, query_hash,
 vs.namespace_or_collection, filter_clause,
 returned_count, returned_tenant_ids_set}

Then add a synchronous assertion in the request path:

assert returned_tenant_ids_set.issubset({session.tenant_id, GLOBAL_TENANT})

This converts a silent leak into a loud 500 — the right failure mode.

Stage 7 — Cache / rerank / postprocessing audit

Walk every component after retrieval and confirm none of them:

• Caches results across tenants under a tenant-agnostic key (e.g. cache_key = hash(query) is a leak; correct is cache_key = hash((tenant_id, query))).
• Returns "fallback" results from a global pool when the tenant pool is empty.
• Logs full retrieved content to a shared monitoring store readable by other tenants' staff.

Reranker / NodePostprocessor / synthesizer can only safely narrow the set; they never restore foreign tenants and never invent context — confirm by reading the code path.

Stage 8 — Ongoing verification

• Add the cross-tenant property test to nightly CI, not just merge gate.
• On schema migration: re-run with the new chunking / embedding model.
• On vendor upgrade: re-read the changelog for filter / namespace semantics (this changes more than vendors admit — see Pinecone disk-based metadata filtering, Qdrant 1.16 tiered multitenancy).

4. 操作模型 (Operation Models)

Format: Trigger / Action / Output / Evidence. Vendor-specific filter syntax canonicalised against current docs (May 2026).

OP-01 EmbedTenantAtIngest

• Trigger: New ingestion pipeline; any system not currently writing tenant_id into metadata.
• Action: Add tenant_id (from authenticated session) to every node's metadata at chunk creation. Validate non-empty before add() / upsert(). Make the field part of the ingestion schema.
• Output: 100% of vectors carry the immutable tenant key; ingestion refuses to write otherwise.
• Evidence: Qdrant Multitenancy docs; Pinecone Implement multitenancy guide; we45 RAG leakage post-mortem.

OP-02 ChooseIsolationPrimitive

• Trigger: Greenfield RAG, or migration from single-tenant.
• Action: Use the cardinality / latency / cost matrix in Stage 1. Default ladder: <100 tenants → namespace/collection per tenant; 100-10k → payload partition + index (is_tenant); >10k → tiered (hot collection + cold archive). Pair with query filter (Principle 3).
• Output: A documented primitive + the one-line justification.
• Evidence: Pinecone Namespaces vs. metadata filtering; Qdrant Multitenancy and custom sharding; Weaviate Multi-tenancy operations.

OP-03 LockTenantSourceToSession

• Trigger: Code review reveals tenant_id = request.json["tenant_id"] or similar untrusted source.
• Action: Bind tenant_id to the authenticated principal (JWT claim, session row, signed header). Pass through one chokepoint (Context / RequestState) — never read user input again downstream.
• Output: A single get_tenant_id(ctx) accessor used everywhere; no string-typed tenant ids floating through function args.
• Evidence: OWASP LLM01 (prompt injection) + LLM06 (sensitive info disclosure); Christian Schneider RAG security: the forgotten attack surface.

OP-04 PineconeFilterAtQuery

• Trigger: Pinecone-backed RAG; multi-tenant.
• Action: One namespace per tenant; and pass filter={"tenant_id": {"$eq": tenant_id}}. Avoid $in lists >10,000 (hard cap).

  index.query(
      namespace=tenant_id,                       # primary isolation
      vector=embedding,
      top_k=8,
      filter={"tenant_id": {"$eq": tenant_id}},  # defence-in-depth
      include_metadata=True,
  )

• Output: Two-layer isolation; delete_namespace(tenant_id) for offboarding.
• Evidence: docs.pinecone.io/guides/index-data/implement-multitenancy, docs.pinecone.io/troubleshooting/namespaces-vs-metadata-filtering.

OP-05 WeaviateTenantHandle

• Trigger: Weaviate-backed RAG.
• Action: Enable multiTenancyConfig(enabled=True) on the collection; create one tenant per customer; pass tenant= on every read/write.

  coll = client.collections.get("Docs").with_tenant(session.tenant_id)
  coll.query.near_vector(vector=emb, limit=8)

  Optionally auto_tenant_activation=True if tenant set is sparse.
• Output: Each tenant on a separate shard; cross-tenant reads physically impossible from a single client handle.
• Evidence: docs.weaviate.io/weaviate/manage-collections/multi-tenancy; Rethinking Vector Search at Scale (Weaviate blog).

OP-06 QdrantIsTenantPayload

• Trigger: Qdrant-backed RAG with ≥hundreds of tenants.
• Action: Single collection; create payload index with is_tenant=True; filter on every search.

  client.create_payload_index(
      collection_name="docs",
      field_name="tenant_id",
      field_schema=models.KeywordIndexParams(
          type="keyword", is_tenant=True))
  
  client.query_points(
      collection_name="docs",
      query=emb, limit=8,
      query_filter=models.Filter(must=[
          models.FieldCondition(
              key="tenant_id",
              match=models.MatchValue(value=session.tenant_id))]),
  )

• Output: Per-tenant sub-indexes co-located on shared shards; scales to millions of tenants.
• Evidence: qdrant.tech/documentation/manage-data/multitenancy/; qdrant.tech/articles/multitenancy/; qdrant.tech/documentation/examples/llama-index-multitenancy/.

OP-07 ChromaWhereFilter

• Trigger: Chroma-backed RAG (often local / smaller scale).
• Action: Filter on every call.

  collection.query(
      query_embeddings=[emb], n_results=8,
      where={"$and": [
          {"tenant_id": {"$eq": session.tenant_id}},
          {"deleted":   {"$eq": False}},
      ]},
  )

  Same where= on get, update, delete.
• Output: Logical isolation. Pair with per-tenant collection for stronger isolation if cardinality permits.
• Evidence: docs.trychroma.com/docs/querying-collections/metadata-filtering; cookbook.chromadb.dev/strategies/multi-tenancy/naive-multi-tenancy/.

OP-08 PgvectorRLS

• Trigger: pgvector-backed RAG inside an existing Postgres app.
• Action: Add a tenant_id column; enable RLS; create a policy.

  ALTER TABLE chunks ENABLE ROW LEVEL SECURITY;
  CREATE POLICY tenant_isolation ON chunks
      USING (tenant_id = current_setting('app.current_tenant')::uuid);
  
  -- per-request, before the ORDER BY embedding <-> $1:
  SET LOCAL app.current_tenant = '...';

  Application can never see other tenants' rows even with SELECT *.
• Output: Database-enforced isolation; survives ORM bugs and SQL typos.
• Evidence: Supabase pgvector multi-tenant RAG references; standard Postgres RLS docs.

OP-09 CrossTenantPropertyTest

• Trigger: Any retrieval code change merges.
• Action:

  def test_cross_tenant_isolation(retriever_factory):
      ingest("alpha", ["alpha-secret about widgets X1, X2"])
      ingest("beta",  ["beta confidential roadmap for Q3"])
  
      # Query as alpha for a topic beta owns
      r_alpha = retriever_factory(tenant="alpha").retrieve("Q3 roadmap")
      assert all(n.metadata["tenant_id"] == "alpha" for n in r_alpha)
      assert len(r_alpha) >= 0   # zero is fine; foreign is not
  
      # And the reverse
      r_beta = retriever_factory(tenant="beta").retrieve("widget X1")
      assert all(n.metadata["tenant_id"] == "beta" for n in r_beta)

• Output: Regression-proof guarantee that the filter is wired through. Gates merge.
• Evidence: Direct application of Principle 1; mirrors red-team test patterns from CSO Online Securing RAG pipelines in enterprise SaaS.

OP-10 RuntimeAssertReturned

• Trigger: Production endpoint serving multi-tenant RAG.
• Action: After every retrieve, before passing to LLM:

  bad = [n for n in nodes
         if n.metadata.get("tenant_id") not in {ctx.tenant, GLOBAL_TENANT}]
  if bad:
      log.critical("cross_tenant_leak", tenant=ctx.tenant, bad=bad)
      raise SecurityError("retrieval boundary violated")

• Output: Silent leak → loud 500; alert fires on the first occurrence.
• Evidence: Defence-in-depth with audit logging; mirrors EchoLeak / CVE-2024-41892 mitigations.

OP-11 LlamaIndexAutoRetriever (advanced, with caveat)

• Trigger: Heterogeneous metadata; tenant + doc_type + date filters; want LLM to derive filter from natural language.
• Action: VectorIndexAutoRetriever with a VectorStoreInfo describing filterable fields. Pin tenant_id server-side — the auto-retriever decides other filters, but tenant_id is always injected from the session.
• Output: Flexible filter selection without exposing tenant to LLM.
• Evidence: LlamaIndex MetadataFilters + AutoRetriever docs; Principle 2 (tenant from session, not from content).

OP-12 NamespaceVsFilterDecision

• Trigger: Greenfield; one tenant per namespace vs payload partition.
• Action: Use the rubric:
  • Cross-tenant queries ever needed (admin search, support tooling)? → payload partition (filter); namespace per tenant blocks cross-namespace queries.
  • Per-tenant deletion / GDPR erasure frequent? → namespace per tenant (delete_namespace is O(1)).
  • Tenant count >10k? → payload partition (namespaces have administrative overhead at scale).
  • Read cost-sensitive on Pinecone? → namespace per tenant (RU billed per namespace size).
• Output: One documented choice + a written rationale.
• Evidence: docs.pinecone.io/troubleshooting/namespaces-vs-metadata-filtering.

5. 困境决策案例 (Dilemma Cases)

Dilemma 1 — Shared embeddings vs per-tenant embeddings

困境: Re-embedding the same public chunk (e.g. shared regulation text) once per tenant doubles cost and complicates updates. But sharing a vector across tenants means it lives in a "global" pool that must be readable by all — opening a path for poisoned global content to reach every tenant.

约束:

• Embeddings are deterministic given (model, text) — no cryptographic per- tenant signal in the vector itself.
• A shared vector with tenant_id="GLOBAL" is by definition in scope for every tenant's filter — it is not isolated, it is intentionally shared.
• Poisoning at ingestion time of shared content propagates to every tenant simultaneously (Slack AI 2024 incident pattern).

决策步骤:

1. Classify content: tenant-private vs shared-public.
2. Shared-public goes to a separate global namespace / collection with write-restricted ingestion (only platform admins, signed source) and is served via a second retrieval call, not by widening the tenant filter.
3. Per-query: retrieve from tenant pool + (optionally) from global pool, union and dedupe at the application layer with explicit provenance tagging in each node's metadata.
4. Never mix them by filter={"tenant_id": {"$in": [tenant, "GLOBAL"]}} — that pattern hides which pool a chunk came from in downstream logs.

结果: Two pools, two retrieval calls, one application-layer merge. Cost deduplicated for public content; private content stays in private isolation; provenance is auditable.

可提取的操作: OP-01, OP-04..08 (per-pool), OP-10.