Langchain Deploy Integration

LangChain Deploy Integration (Python)

Overview

An engineer ships a working LangGraph agent to Vercel. Every non-trivial request returns FUNCTION_INVOCATION_TIMEOUT. The Python runtime on Vercel defaults to a 10-second cap (P35) — a three-tool agent with one RAG round easily runs 20-40s. Local dev never exposed the wall because uvicorn on a laptop has no timeout. Two fixes apply together and each is load-bearing:

// vercel.json — the baseline cap bump (Pro plan max is 60s, Enterprise 900s)
{ "functions": { "api/chat.py": { "maxDuration": 60 } } }

# app/api/chat.py — stream the response so partial output arrives before the cap
from fastapi.responses import StreamingResponse

@app.post("/api/chat")
async def chat(req: ChatRequest):
    async def gen():
        async for chunk in chain.astream(req.input):
            yield f"data: {chunk.model_dump_json()}\n\n"
    return StreamingResponse(gen(), media_type="text/event-stream",
                             headers={"X-Accel-Buffering": "no"})

The maxDuration: 60 raises the Vercel-imposed wall; streaming reduces time-to-first-byte to under a second so the user sees progress even on a 40-second completion. Once the Vercel cap is fixed, the next three walls are: Cloud Run cold starts (5-15s p99 on Python + LangChain — P36), .env secrets leaking via docker exec <pod> env (P37), and SSE streams hanging because Nginx / Cloud Run buffer the final chunk (P46).

This skill walks through a production-grade multi-stage Dockerfile, Cloud Run flags for cold-start mitigation, Vercel maxDuration + streaming, LangServe route mounting with FastAPI lifespan, SSE anti-buffering headers, and Secret Manager via pydantic.SecretStr. Pin: langchain-core 1.0.x, langgraph 1.0.x, langserve 1.0.x. Pain-catalog anchors: P35 (Vercel 10s default), P36 (Cloud Run cold start), P37 (.env leaks), P46 (SSE buffering).

Prerequisites

• Python 3.11+ (3.12 preferred for uvicorn startup speed)
• langchain-core >= 1.0, < 2.0, langgraph >= 1.0, < 2.0, langserve >= 1.0, < 2.0
• fastapi >= 0.110, uvicorn[standard] >= 0.27
• Target platform: gcloud CLI (Cloud Run), vercel CLI (Vercel), or docker (generic)
• For Cloud Run: a GCP project with Secret Manager API enabled
• For Vercel: a project with @vercel/python runtime configured

Instructions

Step 1 — Multi-stage Dockerfile with slim runtime and uvicorn

A multi-stage build keeps the runtime image under 400MB, which cuts Cloud Run cold starts by 2-3 seconds. Use python:3.12-slim as the final stage (not python:3.12 — that base adds ~900MB for dev tooling that never runs in prod).

# syntax=docker/dockerfile:1.7
FROM python:3.12-slim AS builder
WORKDIR /build
RUN pip install --no-cache-dir uv
COPY pyproject.toml uv.lock ./
RUN uv export --format requirements-txt --no-hashes > requirements.txt \
 && pip wheel --wheel-dir=/wheels -r requirements.txt

FROM python:3.12-slim AS runtime
RUN useradd -m -u 10001 app
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/* \
 && rm -rf /wheels
COPY --chown=app:app app/ ./app/
USER app
EXPOSE 8080
ENV PORT=8080 PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
CMD ["sh", "-c", "uvicorn app.main:app --host 0.0.0.0 --port ${PORT} --workers 1"]

Single worker is correct — Cloud Run handles horizontal scale; in-process multi-worker just duplicates LangChain client memory. See Dockerfile and Secrets for the distroless variant and the .dockerignore hardening for .env files.

Step 2 — Deploy to Cloud Run with cold-start mitigation

Python + LangChain + tiktoken + one embedding model imports take 5-15 seconds (P36). At --min-instances=0, every scale-from-zero request eats that as user-facing latency. Paying for one always-on instance is usually cheaper than the lost requests.

gcloud run deploy langchain-api \
  --source=. \
  --region=us-central1 \
  --min-instances=1 \
  --max-instances=20 \
  --cpu=2 --memory=2Gi \
  --cpu-boost \
  --no-cpu-throttling \
  --timeout=3600 \
  --concurrency=80 \
  --set-secrets=ANTHROPIC_API_KEY=anthropic-key:latest,OPENAI_API_KEY=openai-key:latest \
  --service-account=langchain-api@PROJECT.iam.gserviceaccount.com
# --timeout=3600 is the Cloud Run per-request maximum (1 hour) — needed
# because multi-tool LangGraph agents routinely run 1-5 minutes end-to-end.

The load-bearing flags: --min-instances=1 kills cold-start p99 (one always-warm replica costs ~$15/mo and dominates p99 improvement); --cpu-boost doubles CPU for the first 10 seconds; --no-cpu-throttling (CPU-always-allocated billing) keeps astream running between keepalive pings so long LangGraph runs do not stall at tool boundaries; --concurrency=80 matches typical I/O-bound workloads (drop to 10 if embedding large docs in-process).

See Cloud Run Deploy for VPC egress, file secret mounts, revision traffic splitting, and the full cost model.

Step 3 — Vercel Python: maxDuration: 60 + streaming to beat the cap

On Vercel Hobby the max is 10s by default (P35); Pro is 60s, Enterprise 900s. Always set maxDuration explicitly — the default is a trap.

// vercel.json
{
  "functions": {
    "api/chat.py": { "maxDuration": 60, "memory": 1024 }
  }
}

Streaming is not just a UX fix — it is the mitigation for bursts that still exceed maxDuration. Time-to-first-byte under a second keeps the proxy considering the request alive; partial content renders on the client; when the cap finally triggers, the user has already seen most of the answer. The Vercel entrypoint pattern mirrors the Overview snippet above — pair with the SSE headers from Step 5.

Edge Runtime is not an option here — @vercel/edge is JavaScript-only. Anything that imports langchain must run on @vercel/python (serverless, Node-free Python container). See Vercel Python Deploy for env vars vs Vercel Secrets, cold-start profiling, and the serverless vs fluid-compute tradeoff.

Step 4 — LangServe: add_routes + FastAPI lifespan for pool cleanup

LangServe ships typed HTTP routes over any Runnable. The playground path is invaluable in dev but must be disabled in production — it leaks chain topology to anyone who can hit the URL. Mount behind a FastAPI lifespan that closes asyncpg / httpx / Redis pools on revision retirement; on_shutdown fires too late on Cloud Run and connections leak across revisions.

# app/main.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from langserve import add_routes

@asynccontextmanager
async def lifespan(app: FastAPI):
    app.state.chain = build_chain()
    yield
    await db_pool.close()

app = FastAPI(lifespan=lifespan)
add_routes(app, build_chain(), path="/chat",
           enable_feedback_endpoint=False,
           playground_type="chat" if __debug__ else None)  # None = off in prod

See LangServe Patterns for typed input/output schemas, auth middleware, and coexisting with raw FastAPI handlers.

Step 5 — SSE anti-buffering: survive Nginx, Cloud Run, Cloudflare

Nginx, Cloud Run's load balancer, and Cloudflare all buffer responses by default. On SSE, buffering means the client never sees the final end event and LangGraph.astream hangs forever (P46). Two headers plus one response flush fix it:

from fastapi.responses import StreamingResponse

def sse_headers() -> dict:
    return {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache, no-transform",
        "X-Accel-Buffering": "no",          # disables Nginx buffering
        "Connection": "keep-alive",
    }

@app.post("/api/chat/stream")
async def stream(payload: dict):
    async def gen():
        async for event in graph.astream_events(payload, version="v2"):
            yield f"data: {event['data']}\n\n".encode("utf-8")
        yield b"event: end\ndata: [DONE]\n\n"
    return StreamingResponse(gen(), headers=sse_headers())

Cross-reference: this is the same anti-buffering pattern used by langchain-langgraph-streaming (L29) — that skill covers the astream_events event shapes and client reconnection; this skill covers the proxy surface. If you see the stream work locally but hang in prod, the header is missing on the upstream response, not the client.

Step 6 — Secret Manager over .env with pydantic.SecretStr

python-dotenv populates os.environ. Anyone with container access runs docker exec <pod> env and reads every API key in plain text (P37). Mount secrets from Secret Manager on Cloud Run, Vercel Secrets on Vercel; wrap in pydantic.SecretStr so repr(), logs, and tracebacks print ********** instead of the value.

from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=None, extra="ignore")
    anthropic_api_key: SecretStr
    openai_api_key: SecretStr

settings = Settings()  # reads real env vars only; never .env in prod

Cloud Run: --set-secrets=VAR=secret-name:latest (Step 2). Vercel: vercel env add ANTHROPIC_API_KEY production. Never commit .env; add .env* to .dockerignore so it does not enter the build context.

Cross-reference: this skill sets the deployment boundary. langchain-security-basics (P18) covers key-rotation cadence, PII log redaction, and the broader threat model — get the boundary right here, then layer on P18.