Perf Optimization

Performance Optimization Skill

When to Use

• User provides profiling data (pprof, flamegraph, py-spy, Chrome DevTools, Dart DevTools)
• User asks to analyze or optimize performance of a specific component
• A benchmark regression is detected
• After deploying a new feature that touches a hot path

Contents

Usage

The /perf-optimize workflow loads this skill automatically.

• Methodology & patterns: The coordinator reads this file for the core methodology (Profile → Analyze → Optimize → Benchmark), optimization pattern catalog, anti-patterns, and when-to-stop heuristics.
• Dimension scope cards: Read references/perf-dimensions.md to get the scope definition for each dimension (A–F) and the system prompt template for subagent dispatch.
• Report template: Read references/perf-report-template.md when writing the Phase 3 analysis report and updating it with Phase 6 implementation results.
• Language modules: Read languages/{lang}.md for language-specific profiling tools, irreducible floors, and runtime-specific patterns.

Core Methodology

graph LR
    P1[Profile] --> P2[Analyze]
    P2 --> P2b[Opportunity Scan]
    P2b --> P3[Prioritize]
    P3 --> P4[Optimize]
    P4 --> P5[Benchmark]
    P5 --> P6{Improvement?}
    P6 -->|Yes| P7[Verify & Ship]
    P6 -->|No| P3

Step 1: Profile

Collect profiling data using the language-appropriate tool. Load the relevant languages/*.md module for exact commands.

Output: Raw profiling data (CPU profile, heap profile, or trace).

Step 2: Analyze

Read the profile. Focus on these principles (universal across all runtimes):

1. Focus on cum (cumulative): The total resources consumed by a function AND everything it called. This finds the expensive architectural flows.
2. Contextualize flat: Resources consumed by the function itself only. If a runtime function (GC, malloc, syscall) has high flat time, trace it UP the call chain to find the user-land code that triggered it.
3. Ignore runtime noise: Scheduler overhead (runtime.mcall, runtime.systemstack, GC workers) will always appear. Note if GC pressure is high, but don't try to "fix" the scheduler.
4. Separate benchmark artifacts from production cost: Test harness allocations (e.g., httptest.NewRequest, ResponseRecorder) inflate heap profiles but don't exist in production.

Output: Structured analysis document in docs/research_logs/{component}-perf-analysis.md.

Step 2b: Opportunity Scan

Scope: Apply this checklist ONLY within the hot paths identified by the profiler in Step 2. Do NOT scan the entire codebase — that leads to premature optimization. The profiler pointed you at specific modules; now systematically scan those modules for these categories of waste.

Concurrency & Parallelism:

• Sequential I/O calls that could run concurrently (parallel fetch, gather, join)
• Task/goroutine/thread spawn overhead exceeding the work itself
• Locks held across I/O boundaries (database calls, network, file system)
• Unbounded queues or channels causing memory pressure under load
• CPU-bound work running on an async/event-loop runtime instead of a worker pool

Memory & Allocation:

• Heap allocations inside hot loops (new objects per iteration)
• Missing pre-sized collections (growing arrays/maps from zero)
• Unnecessary copies where borrowing, referencing, or copy-on-write suffices
• Temporary objects that could be reused across iterations (buffer pools)
• Large structures passed by value instead of by reference

Data Structures & Algorithms:

• Linear scans (O(n)) where hash-based lookups (O(1)) would work
• Nested iterations creating O(n²) behavior replaceable with single-pass or hash-based approaches
• Missing early returns or short-circuit evaluation skipping unnecessary work
• Redundant sorting of already-sorted data or where ordering is not required
• Inefficient string building (concatenation in loops instead of buffered writes)

Serialization & I/O:

• Full deserialization when only a subset of fields is needed
• Missing buffered I/O on file or network streams
• Repeated serialization of the same unchanged data (cache the serialized bytes)
• Synchronous/blocking I/O on an async runtime
• Excessive debug/trace logging in hot paths without level gating

Caching & Lazy Initialization:

• Repeated expensive computations with identical inputs (regex compilation, template parsing, config loading)
• Missing lazy initialization for rarely-used resources
• Redundant work that could be memoized or pre-computed at startup

Output: Append opportunity scan findings to the analysis document. Each finding must reference the profiler evidence that led to the hot path.

Step 3: Prioritize

Rank fixes by impact/risk ratio:

Rule: If a fix requires more than 1 day AND saves < 20% on the hot path, defer it.

Step 4: Optimize

Implement one fix at a time. For each fix:

1. Write tests FIRST (TDD — Red → Green → Refactor)
2. Implement the fix
3. Add PERF: inline comments explaining the optimization rationale — what the profiler showed and why the new approach is faster. Without these, a future developer may "clean up" the optimization thinking it's unnecessary complexity.
4. Run all existing tests to verify no regression
5. Benchmark immediately
6. Run quality checks (formatter, linter, security scanner)
7. Commit independently using the structured format below

Never batch multiple optimizations into one commit. Each fix must be independently verifiable and revertable.

Commit format:

perf(scope): one-line description

What: <the optimization implemented>
Why: <what the profiler showed — the performance problem it solves>
Impact: <expected improvement, e.g., "Eliminates ~500 allocations per request">
Measurement: <how to verify, e.g., "Run BenchmarkX, compare allocs/op">

Size guidance: Each fix should be a focused, minimal change. If a fix requires > ~100 lines of changes or touches > 3 files, re-evaluate whether it's actually a refactor in disguise — and if so, use the /refactor workflow instead. Performance commits are surgical; architectural restructuring is a separate concern.

Step 5: Benchmark

Compare before/after with the exact same benchmark configuration (same -benchtime, same -count, same machine load). Report:

• ns/op (latency)
• B/op (memory per operation)
• allocs/op (heap allocations per operation)

Step 6: When to Stop

Stop optimizing when any of these are true:

• Remaining CPU is in hardware-optimized assembly (AES-NI, P-256, SIMD) — you cannot beat the hardware
• Remaining allocations are from the language runtime itself (GC, goroutine stacks, HTTP server internals)
• The fix requires a custom implementation of a well-audited library — the security/maintenance risk outweighs the perf gain
• The measured improvement is < 5% and within benchmark noise
• The remaining optimization requires a large architectural refactor (> 3 files, > 100 lines) — defer to a dedicated /refactor session with its own testing and verification

Documenting failures: Record optimizations that were tried but didn't work in the research log (docs/research_logs/{component}-perf-analysis.md). For each failed or skipped optimization, document: (1) what was tried, (2) expected improvement, (3) actual result, and (4) why it didn't work. This prevents future sessions from repeating the same failed experiments. Also note any surprising profiler findings that reveal codebase-specific performance characteristics.

Optimization Pattern Catalog

These are generic, language-agnostic patterns. Apply them when the profiling data shows the corresponding symptom.

Pattern: Result Caching

Symptom: Same expensive computation repeated with identical inputs (crypto verification, JSON parsing, regex compilation).

Fix: Cache results keyed by input hash. Use bounded LRU with TTL to prevent memory exhaustion.

Safety invariant: When caching security-sensitive results (auth tokens, permission checks):

• ALWAYS re-validate expiry/revocation on cache hit
• ALWAYS bound cache size (DoS protection)
• ALWAYS set TTL shorter than the security credential's validity period

Pattern: Pre-allocation

Symptom: High allocs/op from repeatedly constructing the same objects (option structs, config slices, header maps).

Fix: Build the object once at init time, share it read-only across requests. Safe for concurrent use if the object is immutable after construction.

Pattern: Fast-Reject / Short-Circuit

Symptom: Expensive validation path runs even for clearly invalid inputs.

Fix: Add a cheap structural pre-check before the expensive path. Examples: check string length before regex, count delimiters before parsing, check content-type before deserialization.