Coding Trace Analyze

Coding Trace Analyze

Overview

Use this skill to analyze normalized round traces after collection and sanitization. Analysis and validation default to the released DuckDB at trace/syfi_coding_trace.duckdb. JSONL is a pipeline boundary for collection, merging, sanitization, privacy audit, and optional DB materialization; do not make it the default analysis source.

Analyses are organized as artifact experiments. Each experiment is a folder under artifacts/<category>/<experiment>/ containing one analyze/plot script, a README.md that states the question, inputs, and exact metric definitions, and (when run) the generated outputs. Shared logic lives in artifacts/utils/. Validation and audit checks live separately under validators/<category>/<validator>/ and are run by validators/run_all.py.

Categories:

• artifacts/trace_facts/ — headline facts and format conversions (overview_summary, csv_export).
• artifacts/llm_generation/ — input token composition + generation timing (prefix_append_distribution, adjusted_prefix_append, output_append_assignment, token_spindles, output_tokens, generation_time_cdf, append_vs_prefix_latency, timing_fit, timing_feature_ambiguity).
• artifacts/tool_calls/ — tool_latency_distribution, tool_call_counts, tool_time_by_kind, tool_category_distribution, claude_long_tool_calls.
• artifacts/prefix_cache/ — cache hit-rate behavior (cache_hit_ratio, cache_hit_idle_relationship, cache_replay, kv_cache_active_ratio).
• artifacts/human_in_the_loop/ — human_input_wait, user_turn_response_time, user_turn_decomposition.
• artifacts/session/ — session_token_steps, total_input_growth.
• validators/ — integrity/formula/denominator checks such as trace_facts/tool_duplicate_audit, human_in_the_loop/user_turn_response_audit, human_in_the_loop/user_turn_gap_audit, and human_in_the_loop/e2e_formula_check.

First Steps

1. Work from the coding-trace repo root, identified by pyproject.toml, README.md, and the artifacts/ and scripts/ directories.
2. Use trace/syfi_coding_trace.duckdb for normal analysis. Pass --db only to select another already-materialized sanitized trace.
3. Use uv run python ... so matplotlib, numpy, and pillow resolve from the project environment.
4. Keep artifacts and validators separate: plotting/analysis outputs belong under artifacts/; formula, denominator, and integrity checks belong under validators/.
5. Let artifacts/run_all.py derive artifacts/llm_generation/timing_fit/timing_fit_trace.csv from the selected DuckDB. Pass --timing-input only for an intentional external timing CSV override.
6. Read the experiment's README.md first — it documents exactly how that experiment computes its metric (TTFT, effective tool latency, generation time, cache-hit ratio, etc.). The shared definitions are collected in artifacts/utils/README.md.

Quick Summary

uv run python artifacts/trace_facts/overview_summary/analyze.py
uv run python artifacts/trace_facts/overview_summary/analyze.py --json

Reports separate merged/Claude/Codex sections with scope, token, generation-timing, and tool totals. Treat normalized decoding speed as a trace-level proxy from input-ready event to last model-output event, not a serving-engine decode timer. The post-reasoning TPOT estimate and TTFT residual are computed only for rows with exact reasoning-token counts; provider sections without that accounting report these values as null.

Run The Artifact Suite

Use the artifact dispatcher when running all artifact experiments, a category, or scripts with nonstandard output wiring (stdout capture, CSV export, or derived timing CSVs).

uv run python artifacts/run_all.py --list
uv run python artifacts/run_all.py
uv run python artifacts/run_all.py --only tool_calls
uv run python artifacts/run_all.py --only prefix_cache/cache_hit_ratio

The artifact dispatcher defaults to 16 concurrent jobs. Use --jobs 1 for serial runs, --dry-run to inspect commands, and --stop-on-fail when failure should stop launching new experiments. For a full analysis run after sanitization, use:

uv run python artifacts/run_all.py \
  --db trace/syfi_coding_trace.duckdb \
  --log-dir "$TMPDIR/coding_trace_artifact_runlogs"

Run Validators

Use the validator dispatcher for checks that validate trace integrity, metric denominator coverage, and formula assumptions. These scripts write reports under validators/, not under artifacts/.

uv run python validators/run_all.py --list
uv run python validators/run_all.py
uv run python validators/run_all.py --only human_in_the_loop
uv run python validators/run_all.py --only trace_facts/tool_duplicate_audit

For a full validator run paired with the artifact suite, use:

uv run python validators/run_all.py \
  --db trace/syfi_coding_trace.duckdb \
  --log-dir "$TMPDIR/coding_trace_validator_runlogs"

Running An Experiment

Most experiments can also be run directly; outputs land next to the script.

# token input composition
uv run python artifacts/llm_generation/prefix_append_distribution/plot.py
# tool latency distribution, counts, time-by-kind
uv run python artifacts/tool_calls/tool_latency_distribution/plot.py
uv run python artifacts/tool_calls/tool_call_counts/plot.py
# generation-time and human-wait CDFs
uv run python artifacts/llm_generation/generation_time_cdf/plot.py
uv run python artifacts/human_in_the_loop/human_input_wait/plot.py

DB-backed scripts commonly expose --db, -o/--output-dir, --group-by {provider,model,provider_model}, --sample-size, --pair-sample-size, --per-tool-sample-size, --max-groups, --top-tools, --min-tool-calls-for-plot (collapse rare tool names into Other in PNGs; CSV summaries keep full detail), --seed (deterministic, default 42), and --progress-every.

Some experiments retain -i/--input as an explicit JSONL compatibility path. csv_export requires -o, and overview_summary prints text or --json to stdout; the dispatcher handles those output differences while still passing the selected DuckDB.