Backend Architecture

overbae backend map

Single Django app overbae, project-scoped tenancy.

Layout

• overbae/api/ — DRF views/serializers, one module per surface (eval_views.py, datasets.py, optimizer.py, otlp.py, billing.py, mcp.py, …). views.py and serializers.py are the legacy monoliths — new surfaces get their own module. Global exception handler returns {detail, code, error_id}.
• overbae/models/ — split by domain (iam.py, capabilities.py, traces.py, evaluation.py, finetuning.py, datasets.py, optimizer.py, billing.py, …).
• overbae/services/ — business logic; largest subtrees: eval/ (rubric/judging/cascade/runner), datasets/ (Parquet store, landing, cell runner, contract, alignment, diff, use), mcp/ (Streamable HTTP MCP server — procedure in the mcp skill), codebase/, scan/ (AI-surface extraction, footprint matcher, scan application), capabilities/ (identity, lifecycle, graph).
• overbae/tasks/ — Celery tasks, roughly one module per feature; utils/task_lock.py for locking.
• overbae/modal/ — Modal.com GPU workers (vLLM serving, SFT, PII NER).
• overbae/management/commands/ — backfills and syncs.

Celery topology

Five workers, six queues. Workers are resource profiles; queues are fairness classes.

Three constraints set the worker split. Only prefork enforces time_limit and revoke(terminate=True), so every time-limited task routes to batch, landing or interactive. A loaded prefork child costs hundreds of MB, so wide fan-out cannot be prefork. Orchestration holds its own lane because a chord callback stuck behind work never finalises its run. run_eval_run and preload_capability_eval_set route to io because Modal warm-up and judge authoring block for minutes; the import and workshop reconcilers and queue metrics run on control with short expiries and are dropped when it is full.

io_traces is a second queue on the io worker, not a second worker: one worker over both round-robins (kombu's redis default), so an unbounded trace-scoring burst cannot queue ahead of user-started eval scoring.

interactive sets --prefetch-multiplier=1. batch and landing also disable Redis prefetch so busy processes do not reserve waiting work. Landing workers consume only landing; raising bulk concurrency cannot protect import latency when every bulk slot is occupied.

Evaluation preparation uses EvalGenerationWork receipts and a global scheduler row lock. EVAL_MAX_IN_FLIGHT bounds admitted provider work (default 12), with EVAL_MAX_IN_FLIGHT_PER_RUN (default 2) and least-recently-admitted project/run fairness. Waiting samples remain in Postgres; completion and a 30-second control tick replenish available slots. Completion wakeups coalesce through a cache key, so at most one dispatcher waits on the lock. Failed/cancelled parents cannot claim or continue generation. Lost started attempts become explicit unknown outcomes and are not automatically repeated.

Chat HTTP clients enforce one cancellable elapsed deadline across request bodies, SSE heartbeats and retry waits. They close their connections on expiration; a socket read timeout alone is not a total time bound.

Hosted commands come from docker/worker-topology.json. Deployment runs migrations, verifies a healthy landing-only consumer on the new image, then updates workers, beat and API. Landing and interactive workers run with their queue as the Celery hostname, and overbae.worker_health <queue> checks that the consumer reads only its queue and registers that queue's task; a rollout of either waits for that health check. scripts/plan_landing_capacity.py generates the reviewed ECS, IAM, bounded scale-out and CloudWatch queue-age/newly-blocked-import/monitor-health plan. NewlyBlockedImports counts imports blocked in the last 5 minutes by a system cause (queue_timeout, worker_timeout, dispatch_failed, attempts_exhausted); a user-data failure does not raise it. Scale-in requires an explicit idle drain; automatic scale-in is disabled.

Routing lives in CELERY_TASK_ROUTES and must stay in sync with make worker (one process standing in for the whole fleet, so its -Q lists every queue) , docker-compose and the deployment command manifest. tests/test_celery_topology.py enforces it, and asserts no time-limited task lands on a threads lane. Workers hot-restart via watchmedo on .py changes.

Tracing

Span-only: there is no Trace table. A trace is the set of spans sharing a trace_id; the root span has parent_span_id IS NULL. span_id is the primary key; trace_id is an indexed 32-char hex CharField.

Span carries OTel-native fields plus platform linkage:

• Identity: span_id, trace_id, parent_span_id
• Classification: span_type (llm_call/tool_call/retrieval/workflow), operation
• Timing: start_time_ns, end_time_ns, duration_ns — nanosecond ints, no datetime column
• Status: status_code, status_message
• Resource/scope: service_name, resource_attrs, scope_name, scope_version
• Payload: attributes, events, links
• Linkage, filled by process_span: capability (resolved through IdentityAlias, never created), job, iteration
• Ingest time: received_at

Ingest: POST /api/v1/traces (with a /v1/traces compat alias) takes an OTLP protobuf export → parse ResourceSpans → ScopeSpans → Span → flatten → bulk upsert on span_id → process_span per span for linkage.

Span cost has one owner: services/span_pricing.stamp_span_cost runs on every OTLP and connector span before it is stored. A reported cost (any alias, coalesced to genai.cost) is kept; otherwise genai-prices sets genai.cost from model, tokens and cache reads at the span's start time. The span, trace totals and Capability.usage_stats all read that value, and the SDK never prices. Gateway billing is separate on purpose: model_catalog.estimate_cost charges what OpenRouter charged us.

Read surface, backed by SpanViewSet rather than a trace viewset:

• GET /api/traces/ — one row per trace, its head span (Span.trace_heads): the root once it arrives, else the earliest span, so traces stream in as spans land. Each row carries trace_status (completed = root present, live = rootless and recent, interrupted = rootless and quiet past TRACE_SETTLE_SECONDS)
• GET /api/traces/{trace_id}/ — every span sharing that trace_id, plus root summary and trace_status
• GET /api/traces/services/ — distinct service_name; GET /api/traces/models/ — distinct models
• No /api/spans/ route exists.

Query params are span-native (received_at__gte, trace_id, span_type, operation, service_name, has_error).

Evaluation

Two systems share the evaluator engine but never share a judge. ctx["eval_surface"] decides which, and base.judge_module(ctx) is the only place that chooses — nested callers inherit the surface through ctx rather than importing a judge directly.

The default is trace_scoring. rubric_compiler mirrors the split: build_checklist_prompt for generative, build_judge_prompt for trace scoring.

services/eval/context_check.py supplies model-aware advisory estimates for candidate generation and generative judges through REST POST /api/eval-runs/context-check/, run detail and MCP readiness/resources. Readiness accepts the proposed variants; context warnings never change ready, gate launch or exclude training candidates. Unknown limits remain unverified. Estimates include reserved output; before generation, reference-answer size is only a proxy for judge input. Runtime checks use actual prompts and tool history. The shared judge funnel no longer clips the supplied prompt to a fixed character count. Output exhaustion may retry once with a larger absolute budget bounded by published context/output limits and 64k tokens; unknown capacity or a permanent context rejection does not repeat. _judge sub-scores retain attempt budgets, finish reasons, usage, request IDs and costs. Exhaustion/rejection is a technical error, not a quality zero.

services/eval/context_suggestions.py adds preflight-only alternatives from the configured registry, filtered by published context/output and tool/schema support. Judge suggestions avoid the evaluated provider family. Registry order is retained, not sorted by price. cost_basis scopes USD estimates to all checked inputs plus full reserved output for one pass per model/judge; unknown prices are null. Console shows suggestions during setup, never auto-selects, and uses the existing ranked training candidates and their recommended-configuration training estimates for training alternatives. Trained benchmark hosting costs must not borrow base-model token rates. MCP readiness exposes the same evaluation suggestion contract; training catalog and estimates remain available through its training tools.

Training setup puts the judge selector below the eval set. Blank FinetuningJob.eval_judge_model preserves each evaluator's saved choice (including mixed sets); an explicit choice overrides only generative judge/fallback snapshots for this job, never the library or the Jev decision backend. Before/after runs share those frozen snapshots, and sibling baseline reuse includes judge-selection identity. REST context preview and MCP readiness accept judge_model and return judge_models: all registry judge options, annotated with estimated fit across every grader and aggregate USD budget per evaluated model/dataset pass. Unknown or undersized options stay selectable. MCP start_finetune accepts the same immutable eval_judge_model. Console tints judge/benchmark selectors only for estimated limit overflows, mutes undersized options without disabling them, and uses a warning-coloured Start button to open a short risk summary with Start anyway. Unverified, unavailable and pending checks stay neutral and do not open the warning summary. Context issues never require a model change to launch.

Bounded decisions use core/decisions.py: registered Jev on OpenRouter's /systemone endpoint, structured choice questions, validated complete probability distributions, project/contract-scoped caching, a conservative UTF-8 context guard, and shared Redis request/token pacing. Missing configuration, capacity, invalid responses, oversized context or low confidence fall back through services/eval/decisions.py; confidence is not calibrated correctness. Generative question resolution uses closed, required per-question properties compatible with strict provider schemas, preserves original question IDs and the selected judge, and validates coverage and choices before conversion. No evidence is truncated for Jev. config.decision pins backend (jev or generative), model, confidence floor and adapter-policy version; new snapshots and rubric digests include it. The actual served revision and fallback/cost provenance are stored in _decision sub-scores.

Configurable evaluator policies default to generative. Explicit Jev policies cover checklist verdicts, extracted-claim support, per-turn dimensions, cascade step ratings and confident successful numeric behaviour steps; qualify them against representative labelled examples before relying on their scores. Fixed grounding, capability classification and workshop semantic services retain bounded Jev routing. Independent question batches retain accepted answers and generatively resolve only uncertainty or unfinished questions after a later transport failure. Generated choices carry reasoning, not fabricated Jev confidence. Claim verification reuses extracted claims on fallback. Failed/uncertain behaviour steps retain full generative causal diagnosis. Label-only categorical judges on both evaluation surfaces can opt into Jev Choice with declared labels and unchanged label-to-score mappings; uncertain or insufficient evidence falls back. Categorical checklists, behaviour contracts, panels and cascade routes retain generative review. Jev labels carry decision provenance, not generated explanations. Holistic outcomes, session ledgers, claim extraction, rubric authoring and cascade summaries remain generative. Grounding keeps insufficient evidence outside its support/contradiction denominator; lookup and persistence use one policy-aware identity. Combined dataset description, evaluator relevance and rubric authoring use one generative call. Cascade provenance is stored once per batch, unknown costs include retries and holistic stages, and decision latency measures the whole adapter call including failures. Jev is excluded from inference and chat-model pickers. Provider calls in tests are blocked unless explicitly stubbed.

Generated samples snapshot the runner's initial messages and tool schemas after system-prompt injection in trajectory.model_request; metadata.output_start separates context from newly generated messages. This is the initial runner request, not provider chat-template bytes or every replay turn. services/eval/sample_io.py supplies the REST detail and MCP resource with separate input, generated output, and grading reference. Historical samples fall back only to the verified pinned dataset row and label it input_source=dataset, never an exact captured request; absent sources are unavailable. The eval-run MCP resource exposes five bounded sample inspections with grader reasoning.

Trace scoring carves a trace into units in services/eval/units.py (explicit precedence: turn spans > entry_point invocations > key-segment shim > structural root; beside the lattice, a run boundary enclosing turn slices becomes a run-grain execution surface when a run-grain behaviour binds it, and a boundary-less single-function-span trace is an unscorable orphan fragment); services/eval/trace_scoring.py judges the units and services/behaviour/binder.py binds them.

Standalone REST creation and MCP run_evaluation accept the same optional judge_model used by context preview. Blank preserves mixed library choices; explicit overrides are frozen in EvalRun.judge_model and all generative judge/fallback snapshots, including replay judges attached after launch. Existing runs reject judge changes. Deterministic checks and Jev policy stay unchanged. Console New evaluation uses the shared judge selector, context estimates and advisory Start warning; run detail shows snapshot judges. The MCP run resource exposes the override and bounded run_evaluators judge identities.

Capabilities and sync

The agent is the project itself — one graph per project, no table. Capability rows are its nodes (UI: "Capability"; the sidebar's "Agent" is the product).

• A scan never deletes: absence sets status=leftover and the row reactivates in place when the code returns; observed rows (telemetry- or hand-made) are exempt from leftover.
• DELETE /api/capabilities/{id}/ is a soft delete (status=deleted): the row and its data stay, but it is invisible to lists, scans, and identity lookup, so a later scan of the same code mints a fresh row. There is no merge, split, retire, or history.
• Ingest never creates a capability: services/capabilities/identity.lookup resolves ids/names/slugs through IdentityAlias (renames included), an unknown identity leaves the span unbound, and tasks/capability_rebind re-binds the backlog after scans and hand-made rows.
• Wire identity is id-only: ingest binds spans by overmind.capability.id alone; overmind.agent.* attributes are not read; overmind.capability.name is a display label that never resolves. The overmind/<uuid> model alias never changes.
• Discovery is local: /overmind setup runs overmind chassis (AST inventory + call graph), writes overmind_capabilities.json, convert_json_to_toml fills prompt spans, drops fabricated anchors and unverifiable provenance, then stamps trajectory_map[].verified, then overmind sync.
• Toml sync is POST/GET /api/v1/sync (API-key auth). It maps onto Capability (current/leftover), stores system_prompt / eval_metrics / capability_card / eval_matrix in improvement_metadata (eval_matrix is intent metadata only — not materialized into graders), mints Behaviour rows from capability_card.trajectory_map, keeps repo_summary / trace_provider / toml version in Project.settings, writes leftovers back to toml with archived = true, and enqueues the async Default-set preload (Tier-0 card compiler + Tier-1 LLM judges).
• Repository provenance: overmind chassis saves a local .overmind/scan.json receipt before inspection. Conversion verifies branch, commit and a SHA-256 fingerprint of tracked and unignored working-tree files (excluding generated scan files), then persists repository_snapshot in toml. A changed checkout requires a rescan. Sync never recaptures Git state: it stores provenance in Project.settings and stamps last_synced_at separately; a push without provenance clears the prior label. The agent graph and overmind://project/current expose both. Missing Git or a missing scan receipt leaves provenance unknown.

Scoring

Behaviour-keyed. The codebase scan mints Behaviour/BehaviourVersion contracts per capability (services/behaviour/registry.py, re-anchored across rescans, retired never deleted). Trace scoring carves units (services/eval/units.py), binds each as a TaskExecution (services/behaviour/binder.py — unbound units still materialize), and writes Verdict rows plus the feedback_score["trace_scoring"] block the console reads. The session score is a fold over the append-only ConversationEvent ask ledger. API: api/behaviours.py.

Data Workshop

Source → cells → derived versions, edited only through the dataset's own agent. The full model, runner sandbox, agent tools, landing and export contract: data-workshop skill.

Auth and tenancy

Clerk when CLERK_API_SECRET_KEY is set; blank secret is self-hosted local JWT (POST /api/auth/local/ — email + password, create on first use, no verification). Project-scoped tenancy (User ↔ Project via ProjectMembership), no org layer. Everything queryable is filtered by project.

• A guest (User.is_guest, minted by POST /api/auth/guest/) holds one project and can read and claim. GuestJWTAuthentication (api/authentication.py) refuses every other write with guest_upgrade_required; a view opts in with guest_allowed = True — never a permission class or middleware, since a view's own permission_classes replaces the defaults and a guest identity exists only through that token. Guests get no free credits; /demo points at local setup. A claim moves the memberships to the Clerk account and deactivates the guest; tasks/guest_cleanup.py deletes inactive guests and unclaimed ones after 7 days. Guest claim requires Clerk.
• Console: VITE_SELF_HOSTED=true and a blank VITE_CLERK_PUBLISHABLE_KEY skip ClerkProvider and show the local login form; otherwise Clerk.
• Project invites: with Clerk, an invitation email is sent; without Clerk the ProjectInvite row is stored and claimed on first local (or Clerk) sign-in for that email.
• Commercial billing (remaining-credit 402s, Free/Pro quotas, Stripe Checkout) injects when STRIPE_SECRET_KEY is set (overbae/services/billing_provider.py). Ledger charges always run. Empty key → uncapped OSS: spend is recorded and shown, gates and grants no-op.
• API keys are either scope=account (every project the user belongs to) or scope=project with one resourceIds entry. Creating a key with project always mints project scope.

Training preparation

Workshop findings are advisory before model-specific work: incomplete task/evidence/answer/schema reviews, capability mismatches and train/eval overlap produce warnings, not launch blockers. Exact preprocessing checks technical compatibility and does not repair datasets. Fine-tuning validation is read-only, and successful creation freezes the selected train, validation and eval_cell versions atomically. The same pinned eval cell is reused for before/after runs; dataset versions and capability cannot be swapped after creation. Internal loss-validation splitting preserves duplicates and cannot silently clean selected data.

The workshop is model-independent. services/training_preparation.py caches exact Modal preprocessing by training/validation cell fingerprints, model, training type, context length, training stack and the SFT asset code fingerprint. REST POST /api/training-preparations/ and MCP prepare_training_data share the service; MCP returns a training_preparation job receipt readable through get_job and the jobs resource. Preparation uses CPU-only prepare_<train_function> in the matching Modal training image; it does not start GPU training.

Console setup does not submit or await preprocessing and has no per-model preparation cards or workshop review recommendations. run_finetuning requests or reuses the exact artifact after launch, reports preparation as job progress, and starts GPU training only when ready. Explicit REST/MCP preparation remains available.

Modal recommendations use row-length estimates to size training context without treating estimates as compatibility verdicts. Job preparation repeats sizing across the pinned training and enabled validation cells, preserving a larger requested context. If exact tokenization exceeds that context but fits the model's training-type limit, it requests a matching larger preparation and rechecks every row before GPU submission. Exact lengths need no estimate headroom; over-limit rows and other incompatibilities still block. No dataset is rewritten, truncated or filtered. A training-job retry can recover an undersized context through this same path.

TrainingPreparation persists queued/starting/running/ready/incompatible/failed, the remote call ID and a 20-minute deadline. A 15-second controller queues short io observers. Restarts poll the saved call; transport failures retain it. An unacknowledged submission fails closed. Explicit retry of a confirmed failure cancels its saved remote operation before requeueing; uncertain submissions cannot be retried automatically. REST /retry/ and MCP retry_failed use the same guard. Retrying a failed/cancelled training job also retries its cached confirmed-failed preparation before queuing training, unless GPU training already has a remote ID. Ready or in-flight preparations are reused; unsafe failures and incompatible data refuse the retry without resetting job or eval state.

sft_assets/preprocess.py uses the trainer's pretok path, reporting exact token counts, shifted supervised targets, decoded masked previews and incompatible source-row IDs. It rejects over-context rows and zero-supervision examples without truncation or silent deletion. A ready artifact stores token IDs/labels, checksum and tokenizer; upload_dataset only materializes rows present in that artifact. The GPU engine validates context/vocabulary and consumes those tokens without preprocessing them again. Changes to the model/context or data require a matching artifact. Other providers retain their existing provider-side preprocessing; do not label those configurations exact-preflight-ready.

Changes to the SFT assets or preparation functions require modal deploy overbae/modal/modal_sft_worker.py; recreating local Docker services does not deploy Modal code. A worker/code fingerprint mismatch blocks preparation until the matching worker is deployed. Preprocessing infrastructure failures return a safe, actionable error and remain failed, distinct from incompatible rows. Each preparation retains subprocess stdout/stderr at preparations/<id>/preprocess_stdout.log on the SFT volume, including failed attempts; raw diagnostics do not enter user-facing error messages.

Unsloth token accuracy is measured from the final decoder output through the actual output head, in bounded chunks over supervised next-token targets. Fused CE stays enabled; accuracy does not depend on returned logits or a batch/context threshold. Counts are token-weighted across microbatches, with separate training and validation windows; checkpoint recomputation does not count twice. A missing decoder capture fails explicitly rather than silently dropping the metric. A rolling Modal deploy leaves active calls on their existing code; it cannot add metrics to an already-running trainer or reconstruct historical accuracy.