Dynamo Clone Hotpath Audit

Dynamo Clone Hotpath Audit

<!--
SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: CC-BY-4.0
-->

Purpose

Find .clone() calls in Dynamo Rust request, scheduling, KV, block-manager, and runtime hot paths that can be removed without changing ownership semantics. This is an audit-first workflow, not a blanket clone-removal tool.

Default behavior is read-only audit. Do not edit files, commit, or open a PR unless the user explicitly asks to fix, patch, apply, implement, or create an MR/PR. A skill invocation such as $dynamo-clone-hotpath-audit, "audit", "check", or "scan" is not patch permission.

Prerequisites

• Rust source checkout of ai-dynamo/dynamo.
• Python 3.10+ for the inventory script.
• Ability to run targeted Rust validation commands for touched crates.
• Subagent tooling if available. If subagents are unavailable, run the same roles as separate, explicit review passes and say they were not independent.

Instructions

1. Build The Inventory

Run the read-only scanner from the repository root:

python3 .agents/skills/dynamo-clone-hotpath-audit/scripts/clone_inventory.py \
  --only-actionable \
  --limit 80

Use narrower paths when the user names a subsystem:

python3 .agents/skills/dynamo-clone-hotpath-audit/scripts/clone_inventory.py \
  --paths lib/kv-router/src/scheduling lib/llm/src/backend.rs \
  --only-actionable

Treat the scanner as a triage aid. It ranks likely expensive clones but does not prove removability.

2. Split The Audit By Hot Path

Prioritize in this order:

1. per-request LLM paths: lib/llm/src/backend.rs, preprocessor, HTTP/gRPC services, protocol conversion, migration
2. KV routing and scheduling: lib/kv-router/src/scheduling, lib/kv-router/src/sequences, lib/kv-router/src/indexer
3. KV/block manager event paths: lib/llm/src/block_manager, lib/bindings/kvbm/src/block_manager, lib/kvbm-engine/src/offload
4. runtime request/transport paths: lib/runtime/src/component, lib/runtime/src/pipeline, lib/runtime/src/transports
5. tests, examples, debug paths, and benchmark setup only after hot paths

3. Use Independent Review Roles

For non-trivial audits, use subagents. Give each subagent only the inventory slice and relevant source files, not your intended answer.

Required roles:

• candidate finder: identify high-value clones and classify cheap/required ones
• ownership refactorer: propose concrete borrow, move, Arc, or mem::take changes
• correctness adversary: reject changes that alter sharing, extend lock lifetimes, borrow across .await, break spawned task ownership, or make APIs less clear
• validation planner: choose the smallest tests, clippy commands, or benchmarks that prove the touched behavior

If two roles disagree, keep the clone unless you can write down a precise ownership proof and validation plan.

4. Classify Every Candidate

Use these buckets:

• cheap required: Arc, sender, cancellation token, runtime handle, watch receiver, metrics handle, or tracing span clone needed to share ownership
• semantic required: the original value must stay available for a later use, retry, fan-out, or async task
• cold/test-only: outside production hot paths
• removable move: value is cloned immediately before its final ownership use
• removable borrow: callee does not need ownership and can accept &T, &str, slice, or iterator
• structural refactor: requires changing data layout, e.g. Vec<T> to Arc<[T]>, or changing an API family

Do not patch candidates in the first three buckets.

5. Patch In Small Batches

Only run this section when the user explicitly asks for fixes or a patch. If the user only asked for an audit, stop after the report and list recommended patch batches as follow-up work.

Keep each patch batch narrow: one subsystem or one repeated pattern. Avoid a single repository-wide clone cleanup PR unless the user explicitly asks for it.

Preferred fixes:

• move a value when the clone is immediately consumed and not used afterward
• pass &T, &str, &[T], or an iterator when the callee only reads
• consume batches with into_iter() instead of indexing and cloning
• extract small Copy fields before moving a large event
• use std::mem::take only when leaving the source value empty is part of the intended semantics
• use Arc only when shared ownership is semantically right, not just to avoid borrow-checker work

Do not:

• remove clones that cross tokio::spawn, channel send, callback storage, or task lifetime boundaries without proving ownership
• extend mutex/RwLock guard lifetimes to avoid a clone
• borrow across .await unless the borrow is local and compiler-verified
• trade a clear cheap clone for a confusing lifetime-heavy API
• make public APIs borrow data whose ownership was intentionally independent