Routing Architecture

Routing Architecture Guide

Overview

The router (src/core/router/) selects among deployments — concrete provider+model pairs registered via Router::add_deployment / Router::set_model_list — using one of 7 strategies. There is no per-strategy router struct, no Router trait, and no create_router factory: strategy dispatch is a match on the RoutingStrategy enum calling free functions in src/core/router/strategy_impl.rs.

Key Design Principles

• Snapshot isolation: deployments, the model index, and aliases live in an immutable RoutingSnapshot, published through ArcSwap (src/core/router/unified.rs:61,308). Readers load one generation lock-free; writers clone-modify-store under a parking_lot::Mutex that only serializes writers (unified.rs:312,379-398).
• Atomic deployment state: per-deployment runtime state (DeploymentState) is plain atomics with Relaxed ordering (deployment.rs:210-252); RoundRobin uses a DashMap<String, AtomicUsize> of per-model counters (unified.rs:321).
• Health-aware: candidates are filtered by cooldown, health status, parallel limits, and RPM/TPM limits before a strategy picks among them.
• Fallback chains: the built-in execution path tries the model-keyed General fallback list after retries are exhausted. Typed context-window, content-policy, and rate-limit lists require explicit caller lookup/wiring today.

Selection Flow

Entry point Router::select_deployment_lease (selection.rs:95) delegates to select_deployment_matching (selection.rs:226). The real flow:

1. Load the current snapshot; resolve model aliases via resolve_model_name (max MAX_ALIAS_HOPS = 16 hops, unified.rs:26).
2. Look up the resolved model in snapshot.model_index to get candidate DeploymentIds.
3. One filter pass builds Vec<RoutingContext> (strategy_impl.rs:17), skipping deployments that are in cooldown (is_in_cooldown), unhealthy (is_healthy), at their max_parallel_requests limit, or at their rpm_limit / tpm_limit (selection.rs:287-344). Each context copies weight, priority, active_requests, tpm_current/tpm_limit, rpm_current/rpm_limit, avg_latency_us.
4. Router::select_from_routing_contexts dispatches on the configured strategy (selection.rs:195-224):

match strategy {
    RoutingStrategy::SimpleShuffle => strategy_impl::weighted_random_from_context(routing_contexts),
    RoutingStrategy::LeastBusy     => strategy_impl::least_busy_from_context(routing_contexts),
    RoutingStrategy::UsageBased    => strategy_impl::lowest_usage_from_context(routing_contexts),
    RoutingStrategy::LatencyBased  => strategy_impl::lowest_latency_from_context(routing_contexts),
    RoutingStrategy::PriorityBased => strategy_impl::lowest_priority_from_context(routing_contexts),
    RoutingStrategy::RateLimitAware => strategy_impl::rate_limit_aware_from_context(routing_contexts),
    RoutingStrategy::RoundRobin    => strategy_impl::round_robin_from_context(
        model_name, routing_contexts, round_robin_counters),
}

5. The winner is reserved by try_reserve_deployment — an atomic increment of active_requests (CAS loop when max_parallel_requests is set). If another caller wins the last slot, that candidate is removed from the contexts and selection retries (selection.rs:378-415).
6. Returns a DeploymentLease; dropping it decrements active_requests (RAII release, selection.rs:64-70). The deprecated ID-returning select_deployment still exists but converts the lease to an ID without release-on-drop.

Routing Strategies

Enum RoutingStrategy (src/core/router/config.rs:22-39) serializes as snake_case (simple_shuffle, round_robin, least_busy, latency_based, priority_based, usage_based, rate_limit_aware); PriorityBased also accepts the serde alias "cost_based".

1. SimpleShuffle (runtime default)

Weighted random selection: draws a point in 0..total_weight and walks candidates until the cumulative weight covers it; uniform random when total weight is 0 (weighted_random_from_context, strategy_impl.rs:56-85). Weights come from DeploymentConfig.weight (default 1).

Use when: general traffic where deployments have different capacity.