Public QA Chatbot

Public Q&A Chatbot - Best Practices

A comprehensive skill for building unauthenticated, public-facing Q&A chatbot widgets on marketing sites, conference pages, documentation portals, and similar contexts where you need to serve anonymous visitors while controlling cost and abuse.

Use this as a design and review catalog, not a requirement to add every listed service or subsystem. Select controls for the public exposure, cost, privacy, and UX risks created by the requested change. Server-side secret isolation, server-authoritative enforcement of abuse limits, and safe handling of user content are invariants where those boundaries are touched. Caching, vector search, tracing, virtualization, voice, and multi-layer infrastructure are recommendations or opportunities unless the requested scale or failure mode makes one necessary. Stop when the named widget behavior is usable and its concrete public-abuse/cost risks are bounded; report adjacent improvements as follow-ups.

Distilled from a production implementation powering the AI Engineer Europe 2026 conference chatbot, with additional chat-scroll lessons from TanStack Virtual's chat guidance and agentic retrieval lessons from Mintlify's virtual filesystem assistant. See MINTLIFY_VIRTUAL_FILESYSTEM.md for a clean markdown reference version of the Mintlify pattern.

For a runnable React/TanStack Virtual demo of long chat scroll behavior plus an expanded bottom command shelf, use assets/vite-react-tanstack-chat-demo. The demo includes hover/double-click message controls, subtle token/latency stats, tool-call and multimodal examples, assistant response variants via left/right swipe, and a Realtime voice capture strip with live transcription and an audiogram.

Image: Public Q&A chatbot demo with live voice transcription and bottom command shelf

Run the demo locally:

cd assets/vite-react-tanstack-chat-demo
npm install
npm run dev -- --port 5179

To exercise the Realtime voice path, start the dev server with OPENAI_API_KEY set. Keep the standard API key on the server side only; the browser should receive an ephemeral Realtime client secret.

When to use this skill

• Embedding a chatbot widget on a public website (no user login required)
• Answering questions from a known FAQ / knowledge base
• Serving anonymous visitors with LLM-powered responses
• Needing to protect against abuse, cost overruns, and API quota exhaustion
• Building a constrained Q&A bot (not a general-purpose assistant)
• Reviewing a public chatbot's widget UX, streaming behavior, scroll anchoring, or history loading

Tech stack choices

This skill is written to be tech-agnostic. The reference implementation uses the stack below, but each component is swappable:

Do not require virtualization for every public FAQ widget. A short, bounded chat can stay as a simple DOM list. Reach for a virtualized chat list when conversations can grow long, rows have dynamic heights, older history prepends, or streaming output makes scroll anchoring fragile. When using React and a virtualized list is justified, prefer TanStack Virtual's chat support over custom scroll math.

Do not require agentic document browsing for every public FAQ widget either. Plain RAG is sufficient for short, stable FAQs. Add a virtual docs filesystem when answers live across multiple pages, users ask for exact syntax, docs have a meaningful hierarchy, or top-k retrieval often misses the section an expert would grep for.

1. Rate Limiting

Multi-layer rate limits

Apply limits at multiple granularities to prevent abuse:

• Per-turn: Cap messages per conversation (e.g. 9 turns/session)
• Per-visitor per day: Cap sessions per IP per day (e.g. 15/day)
• Global per day: Cap total sessions across all visitors (e.g. 3000/day)

// Example constants
const LIMITS = {
  turnsPerSession: 9,
  sessionsPerVisitorPerDay: 15,
  globalSessionsPerDay: 3000,
};

Use distributed rate limiting in production

In-memory rate limiting resets on every serverless cold start and isn't shared across instances. Use a distributed store for production:

Upstash Redis (reference):

import { Ratelimit } from "@upstash/ratelimit";
import { Redis } from "@upstash/redis";

const redis = new Redis({ url: REDIS_URL, token: REDIS_TOKEN });
const limiter = new Ratelimit({
  redis,
  limiter: Ratelimit.slidingWindow(15, "1 d"), // 15 per day
  prefix: "chatbot:visitor",
});
const { success } = await limiter.limit(clientIp);

Alternatives:

• Cloudflare Rate Limiting - built into Cloudflare Workers, no external DB needed
• Arcjet - drop-in rate limiting SDK with bot detection
• AWS WAF - rate-based rules at the edge
• Self-hosted Redis - ioredis + custom sliding window logic

Always keep an in-memory fallback for local development:

const useDistributed = !!redisUrl && !!redisToken;
if (!useDistributed) {
  // Fall back to in-memory Map for local dev
}

Server-authoritative counting

Never trust client-reported turn counts or session flags. The server must count turns from the messages array itself:

// Server counts turns - never trust client-reported values
const userTurnCount = messages.filter(m => m.role === "user").length;
const isNewSession = userTurnCount <= 1;

Session counting timing

Only increment the session counter after the server confirms a successful response, not when the user submits. This prevents phantom session counts from failed requests, network errors, or aborted streams:

// Client-side: count after first assistant response arrives
useEffect(() => {
  const hasAssistantMessage = messages.some(m => m.role === "assistant");
  if (hasAssistantMessage && !sessionCounted.current) {
    sessionCounted.current = true;
    incrementSessionCount();
  }
}, [messages]);

Non-new session handling

When a request is not a new session (i.e. a follow-up turn in an existing conversation), skip daily session counter increments entirely. Only the first turn of a conversation should count as a "session" for rate limiting purposes:

if (!isNewSession) {
  return { allowed: true }; // Skip session counting for follow-up turns
}

BYOK (Bring Your Own Key) fallback

When rate-limited, let users input their own API key to continue chatting. This turns abuse into the user's own cost while preserving good UX:

// Skip rate limiting when user provides their own key
if (!userApiKey) {
  const limit = await checkRateLimit(ip, turnCount, isNewSession);
  if (!limit.allowed) {
    return res.status(429).json({ error: limit.reason, rateLimited: true });
  }
}
const apiKey = userApiKey || serverKey;

Provide a direct link to obtain a key (e.g. https://aistudio.google.com/apikey for Gemini, https://platform.openai.com/api-keys for OpenAI).

2. Security

Origin validation

Check the Origin or Referer header against an allowlist. This prevents cross-site request abuse where third parties embed scripts that burn your API quota:

const origin = req.headers.origin ?? req.headers.referer ?? "";
const allowedHosts = ["localhost", "yourdomain.com", "vercel.app"];
if (origin && !allowedHosts.some(h => origin.includes(h))) {
  return res.status(403).json({ error: "Forbidden" });
}

Note: Substring matching (origin.includes(h)) is acceptable for v1 but could theoretically match crafted domains. For stricter validation, parse the URL and compare the hostname.

Input size limits

Cap both the number of messages and individual message length to prevent token-stuffing attacks that run up your LLM bill:

const MAX_MESSAGES = 10;
const MAX_MESSAGE_LENGTH = 2000;

const trimmedMessages = messages.slice(-MAX_MESSAGES).map(m => ({
  ...m,
  parts: m.parts.map(p =>
    p.type === "text" && typeof p.text === "string"
      ? { ...p, text: p.text.slice(0, MAX_MESSAGE_LENGTH) }
      : p
  ),
}));

Also limit model output: maxOutputTokens: 500 for short Q&A answers.

Validate all parameters

Never trust as casts for user-supplied values. Validate against a known set:

const VALID_PAGES = new Set(["europe", "home", "worldsfair"]);
if (!VALID_PAGES.has(page)) {
  return res.status(400).json({ error: "Invalid page parameter." });
}

Access-prune retrieval surfaces

For documentation-backed chatbots, access control must happen before retrieval, not after answer generation. If the bot exposes semantic search, exact search, or a virtual docs filesystem, apply the same visibility filter to every surface:

• Exclude unpublished, draft, internal, customer-only, or role-gated pages before building any path tree the model can browse.
• Apply the same filter to vector, keyword, and chunk queries. Do not rely on hiding paths in the UI while leaving chunks searchable.
• Prefer omitting inaccessible paths entirely. A model should not be able to mention "there is an internal billing page, but you cannot access it."
• Include isPublic, groups, tenantId, docsVersion, or equivalent metadata with indexed chunks so filters are cheap and testable.

Safe error handling

• Never leak raw SDK error strings to the client (may contain API keys from BYOK)
• Never log full error objects (may contain sensitive data)
• Return generic error messages:

} catch {
  console.error("Chat API error");
  return res.status(500).json({
    error: "An error occurred processing your request. Please try again.",
  });
}

IP resolution on serverless platforms

Use the platform's trusted headers. On Vercel: x-real-ip > x-vercel-forwarded-for > x-forwarded-for. The standard x-forwarded-for is spoofable by clients.

Alternatives:

• Cloudflare: CF-Connecting-IP
• AWS ALB/CloudFront: X-Forwarded-For (first IP is trustworthy when set by AWS)
• Fastly: Fastly-Client-IP

Disable non-text modalities

If you only need text responses, explicitly restrict the model:

const model = provider("gemini-3.1-flash-lite", {
  responseModalities: ["TEXT"], // Gemini-specific
  // For OpenAI: modalities: ["text"]
});

Also state "text-only assistant" in the system prompt as a defense-in-depth measure.

3. Cost Optimization

Semantic caching

Use vector similarity search to cache and reuse responses for semantically similar questions. Most effective for FAQ-style chatbots where users ask the same questions in different words.

Upstash Vector (reference):

import { Index } from "@upstash/vector";

const vectorIndex = new Index({ url: VECTOR_URL, token: VECTOR_TOKEN });

// Lookup: check cache before calling LLM
const results = await vectorIndex.query({
  vector: await getEmbedding(question),
  topK: 1,
  includeMetadata: true,
  filter: `page = '${page}'`,
});
if (results[0]?.score >= 0.92 && results[0]?.metadata?.answer) {
  return results[0].metadata.answer; // Cache hit - skip LLM call
}

// Store: cache after LLM responds (fire-and-forget)
void vectorIndex.upsert({
  id: `cache-${Date.now()}`,
  vector: embedding,
  metadata: { question, answer, page, cachedAt: Date.now() },
});

Key decisions:

• Similarity threshold: 0.92+ to avoid returning wrong cached answers. Lower values increase hit rate but risk incorrect responses.
• Embedding dimensions: 128 dims is sufficient for FAQ similarity and cheaper to compute/store than full 768/1536/3072.
• Cache scope: Cache first-turn questions only (highest hit rate, simplest implementation).
• TTL: 7 days is reasonable; stale answers are better than no cache.

Alternatives:

• Pinecone - managed vector DB with metadata filtering
• pgvector - if you already have PostgreSQL
• Cloudflare Vectorize - edge-native, pairs with Workers
• Qdrant/Weaviate - self-hosted or cloud, richer query capabilities

Cache TTL enforcement

Always store a cachedAt timestamp in cache entry metadata. On lookup, reject entries older than your TTL (e.g. 7 days). This prevents stale answers from persisting indefinitely, especially when FAQ content changes:

const CACHE_TTL_MS = 7 * 24 * 60 * 60 * 1000; // 7 days
if (Date.now() - result.metadata.cachedAt > CACHE_TTL_MS) {
  // Stale - treat as cache miss
}

Stream protocol consistency for cache hits

When returning a cached response, use the same streaming protocol as live LLM responses. Don't switch to a different response format (e.g. manual Data Stream Protocol vs. UI Message Stream). Inconsistent formats cause client-side parsing errors and broken UX:

// BAD: different format for cache hits
res.write(`0:${JSON.stringify(cachedText)}\n`); // Manual Data Stream Protocol
// GOOD: same format for both paths
const stream = createUIMessageStream({ /* ... */ });
pipeUIMessageStreamToResponse(stream, res);

Optimized exact search

For docs assistants that expose grep-style tools, avoid scanning every page or chunk over the network. Use a two-stage exact-search path:

1. Coarse filter: ask the document database for pages whose metadata or text might contain the fixed string or regex. Use datastore-native filters where available, such as $contains, full-text indexes, trigram search, or metadata filters by section/path.
2. Bulk prefetch: fetch all candidate chunks for the matching pages in one batch, sorted by page and chunk_index.
3. Fine filter: run exact string or regex matching in memory and return only final hit paths/snippets.
4. Cache: store prefetched page chunks by { path, docsVersion } so repeated grep/cat workflows do not hit the database twice.

Log candidate count and final hit count. If the coarse filter returns too many pages, ask the model to narrow the query instead of silently running an expensive full-corpus scan.

FAQ list view

Offer a browsable FAQ list alongside the chat interface. This serves users who have common questions without making any LLM calls at all:

// Structured FAQ data for UI rendering
export const FAQ_QUESTIONS: Array<{
  category: string;
  question: string;
  answer: string;
}> = [
  { category: "Ticketing", question: "Can I get a refund?", answer: "Yes, per our refund policy..." },
  // ...
];

Organize by category with expandable sections. Clicking a question can either show the pre-written answer directly or send it to the chat for a more detailed LLM response.

Use the cheapest sufficient model

For a constrained Q&A chatbot, you rarely need the most powerful model:

Short output limits

Set maxOutputTokens to the minimum needed (e.g. 500 tokens for 2-4 sentence answers). This caps cost per request and keeps responses concise.

Context caching

Pre-build and cache the system prompt context at module level. This avoids re-computing expensive string concatenations on every request:

let cachedContext: Record<string, string> | null = null;

function buildContext(): Record<string, string> {
  if (cachedContext) return cachedContext;
  // ... expensive computation ...
  cachedContext = result;
  return cachedContext;
}

4. Observability

Trace every LLM call

Instrument all LLM calls with input/output, latency, token usage, and cost. This is essential for monitoring abuse, debugging, and cost tracking.

Braintrust (reference):

import { initLogger, wrapAISDK } from "braintrust";
initLogger({ projectName: "my-chatbot", apiKey: BRAINTRUST_API_KEY });
const { streamText } = wrapAISDK(ai); // Auto-traces all calls

Alternatives:

• Langfuse - open-source, self-hostable, supports OpenAI/Anthropic/custom
• Helicone - proxy-based, zero-code integration
• LangSmith - if using LangChain
• OpenTelemetry - vendor-neutral, export to Datadog/Honeycomb/Grafana
• Datadog LLM Observability - if already using Datadog

Log semantic cache hits

Track cache hit rates to understand cost savings and tune the similarity threshold. A cache hit is a "free" response that saved an LLM call.

Trace retrieval tool calls

Trace retrieval tools separately from LLM calls. For semantic search, exact search, and virtual filesystem tools, log:

• Tool name, query/pattern, requested path, and docs version.
• Latency, cache hit/miss, database round trips, chunks fetched, candidate count, and final result count.
• Whether the model escalated from broad semantic search to exact grep/cat/ls exploration.
• User-visible outcome signals such as cited-answer rate, "I don't know" rate, thumbs up/down, and handoff/escalation rate.

This tells you whether agentic retrieval is improving answer quality or just adding cost and latency.

Don't log sensitive data

Avoid logging full error objects, API keys, or user PII. Log just enough to debug (error type, status codes, IP hashes).

5. UX Patterns

Markdown rendering

Enable markdown in chat responses and instruct the model to use it via the system prompt:

You may use markdown formatting in your responses when appropriate:
- Use **bold** for emphasis on key information like dates, prices, or venue names
- Use [links](url) when referencing websites
- Use bullet points for lists of speakers, sessions, or options
- Keep formatting light and readable

React: react-markdown + remark-gfm Vue: vue-markdown-render Vanilla JS: marked or markdown-it

Draggable and resizable window

Let users reposition and resize the chat window. Persist geometry to localStorage so it survives page reloads. Clamp positions to viewport bounds:

const newX = Math.max(0, Math.min(
  e.clientX - dragOffset.x,
  window.innerWidth - geometry.width
));

Streaming responses

Always stream responses for perceived speed. Use your SDK's streaming API rather than waiting for the full response. The first token appearing quickly matters more than total latency.

For the UI, update one in-progress assistant message as tokens arrive. Do not append a new message row per token. Token-level rows are expensive, break transcript semantics, and make scroll anchoring harder.

Chat scroll and virtualization

Public Q&A widgets often start short, so a plain scroll container is fine until there is evidence it is not. Add virtualization when the widget can hold long histories, rich markdown, tool results, images, code blocks, history pagination, or token-streaming messages that grow in height.

When virtualization is warranted, recommend TanStack Virtual's chat support for React implementations, but keep it optional and swappable. The important lessons are the scroll contracts:

• Treat chat as an end-anchored reverse feed, not a normal top-anchored list.
• Keep message data in normal chronological order; avoid flex-direction: column-reverse, inverted transforms, and hand-maintained scrollTop += delta bookkeeping.
• Use stable message IDs as row keys. Index keys cannot preserve position after prepending older history.
• Loading older history should prepend messages with ordinary array updates, such as setMessages((current) => [...olderMessages, ...current]).
• Follow appended messages only when the user was already near the latest message. If the user scrolled up to read history, incoming output must not yank them back to the bottom.
• Use an explicit "near latest" threshold, e.g. about 80px, rather than exact-bottom checks that are brittle across browsers and dynamic heights.
• Expose a "Latest" or "Jump to bottom" affordance when the user is away from the end.
• Dynamic row heights are the default for real chat. Markdown, links, code, tool output, and streamed text should be measured or allowed to reflow without overlap.
• Prefer instant/auto follow for high-frequency token streaming. Smooth scroll can look nice for discrete appends, but validate it because animation targets can fight dynamic measurement.
• Keep pagination cursors, hasMoreHistory, loading flags, and request dedupe in app state. The virtualizer should receive the current ordered message array, not own data fetching.

TanStack Virtual maps these lessons to anchorTo: 'end', followOnAppend, scrollEndThreshold, stable getItemKey, measureElement, isAtEnd(), getDistanceFromEnd(), and scrollToEnd(). These APIs are useful defaults, not a hard dependency.

Bottom command shelf

Do not treat the composer as only a textbox. For AI applications, the bottom of the screen is valuable thumb-reachable space for the actions users need while forming a prompt: attach, tools, model, voice, send, mode, reasoning depth, runtime context, and tool launchers.

Use a progressive bottom command shelf when the app has enough controls to justify it:

• Keep the default composer compact: input, add/attach, tools toggle, model chip, mic, and send.
• Expand into a bottom sheet for secondary controls rather than putting all controls in the default composer.
• Keep send one tap away in both compact and expanded states.
• Keep the main canvas visually calm; let the bottom shelf become the command plane.
• Show mode and execution state as compact chips, e.g. Plan / Build, effort level, device/project/branch, and budget or usage.
• Place tool launchers in the expanded shelf when they are likely to be used mid-prompt, e.g. terminal, file search, web search, docs, or attachments.
• Add an explicit close/collapse affordance above the expanded shelf so the user can reclaim vertical space.
• Avoid this pattern for simple public FAQ widgets with no tools or settings. For constrained Q&A, a compact composer plus FAQ chips is often enough.
• Reserve layout space for the compact composer, then let expanded toolbar controls move upward as an overlay. Opening tools should not resize, jump, or re-anchor the chat transcript.
• Test keyboard open/close, safe-area insets, shelf expand/collapse, message streaming, and history reading with the shelf in both states.