LLM Integration

LLM Integration

Patterns for integrating LLMs into production applications: tool use, streaming, local inference, and fine-tuning. Each category has individual rule files in rules/ loaded on-demand.

Quick Reference

Total: 20 rules across 7 categories

Quick Start

# Function calling: strict mode tool definition
tools = [{
    "type": "function",
    "function": {
        "name": "search_documents",
        "description": "Search knowledge base",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Search query"},
                "limit": {"type": "integer", "description": "Max results"}
            },
            "required": ["query", "limit"],
            "additionalProperties": False
        }
    }
}]

# Streaming: SSE endpoint with FastAPI
@app.get("/chat/stream")
async def stream_chat(prompt: str):
    async def generate():
        async for token in async_stream(prompt):
            yield {"event": "token", "data": token}
        yield {"event": "done", "data": ""}
    return EventSourceResponse(generate())

# Local inference: Ollama with LangChain
llm = ChatOllama(
    model="deepseek-r1:70b",
    base_url="http://localhost:11434",
    temperature=0.0,
    num_ctx=32768,
)

# Fine-tuning: QLoRA with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B",
    max_seq_length=2048, load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(model, r=16, lora_alpha=32)

Function Calling

Enable LLMs to use external tools and return structured data. Use strict mode schemas (2026 best practice) for reliability. On Claude keep the tools array stable across turns so the prompt cache holds; add tools mid-session with an inline tool_addition (beta inline-tools-2026-09-15, Claude API only, available on Sonnet 5.5 but not on Sonnet 5) instead of swapping subsets (https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages#mid-conversation-tool-changes). For large tool sets, use a non-deferred tool search tool with defer_loading: true on the rest, never defer_loading alone (a request cannot have every tool deferred). Forced tool_choice (any or a named tool) returns a 400 on Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1; use auto with strict: true tools. Validate all inputs with Pydantic/Zod, and return errors as tool results. <!-- model-recency-ok: per-model contrast, Sonnet 5 lacks what Sonnet 5.5 has (platform docs 2026-09-28) -->

• calling-tool-definition.md -- Strict mode schemas, OpenAI/Anthropic formats, LangChain binding
• calling-parallel.md -- Parallel tool execution, asyncio.gather, strict mode constraints
• calling-validation.md -- Input validation, error handling, tool execution loops

Streaming

Deliver LLM responses in real-time for better UX. Use SSE for web, WebSocket for bidirectional. Handle backpressure with bounded queues.

• streaming-sse.md -- FastAPI SSE endpoints, frontend consumers, async iterators
• streaming-structured.md -- Streaming with tool calls, partial JSON parsing, chunk accumulation
• streaming-backpressure.md -- Backpressure handling, bounded buffers, cancellation

Local Inference

Run LLMs locally with Ollama for cost savings (93% vs cloud), privacy, and offline development. Pre-warm models, use provider factory for cloud/local switching.

• local-ollama-setup.md -- Installation, model pulling, environment configuration
• local-model-selection.md -- Model comparison by task, hardware profiles, quantization
• local-gpu-optimization.md -- Apple Silicon tuning, keep-alive, CI integration

Fine-Tuning

Customize LLMs with parameter-efficient techniques. Fine-tune ONLY after exhausting prompt engineering and RAG. Requires 1000+ quality examples.

• tuning-lora.md -- LoRA/QLoRA configuration, Unsloth training, adapter merging
• tuning-dataset-prep.md -- Synthetic data generation, quality validation, deduplication
• tuning-evaluation.md -- DPO alignment, evaluation metrics, anti-patterns

Context Optimization

Manage context windows, compression, and attention-aware positioning. Optimize for tokens-per-task.

• context-window-management.md -- Five-layer architecture, anchored summarization, compression triggers
• context-caching.md -- Just-in-time loading, budget scaling, probe evaluation, CC 2.1.32+

Evaluation

Evaluate LLM outputs with multi-dimension scoring, quality gates, and benchmarks.

• evaluation-metrics.md -- LLM-as-judge, RAGAS metrics, hallucination detection
• evaluation-benchmarks.md -- Quality gates, batch evaluation, pairwise comparison

Prompt Engineering

Design, version, and optimize prompts for production LLM applications.

• prompt-design.md -- Chain-of-Thought, few-shot learning, pattern selection guide
• prompt-testing.md -- Langfuse versioning, DSPy optimization, A/B testing, self-consistency
• prompt-react-pattern.md -- ReAct loop for tool-using agents, thought-action-observation format
• prompt-optimization.md -- Token reduction, cost optimization, model tiering, prompt spec format