Skill Optimization Study

Skill optimisation study (MPS MCP)

A repeatable procedure for finding where agents waste turns and tokens when driving MPS through mps_mcp_*, and for choosing the cheapest fix. This skill is self-sufficient: the 2026-09 study documents (plugins/mcp-tools/docs/skill-script-automation-study.md and its runbook) are history and evidence, not required reading. Scripts and scenario prompts live in plugins/mcp-tools/study/; if that directory is removed, move scripts/ here and the prompts into assets/.

Conventions used below

Agent shells reset the working directory per call, so every command uses absolute paths through two variables — set them at the start of each Bash call (or export them in a wrapper script):

STUDY=/Users/vaclav/work/MPS/myMPS-fix/plugins/mcp-tools/study   # adjust to the checkout
RUNS=$HOME/MPSProjects/mcp-study/runs                            # evidence dir, outside the repo

$RUNS/inventory.json is a load-bearing name: run_worker.sh records its sha in every run's meta.

Roles

• Observer (this session, Opus-class): orchestrates, never performs the MPS task, never tells workers they are measured, evaluates results read-only, writes the report. Owns the whole lifecycle: synthesizes empty projects (scripts/new_study_project.py), opens them via CLI (mps-project-management), closes them with mps_mcp_close_project, and starts, restarts and shuts MPS down with scripts/mps_control.sh. Before each swap, tell the user the absolute path about to close (if any) and the absolute path about to open — do not ask them to perform the swap and do not wait for approval.
• Workers: headless CLI processes (claude -p or junie --task), one per (scenario, model, run), launched by study/scripts/run_worker.sh. Evidence = their transcript + the server call log.
• Human: answers the gate questions, approves pushes, and dismisses MPS dialogs when a call returns MODAL_BLOCKED or a close/exit hangs on a confirmation. Does not open, close or create projects and does not restart MPS — those are observer actions now.

Gate questions to ask before starting (use them verbatim)

Before question 2, run python3 $STUDY/scripts/list_worker_models.py and present models as a multi-select. The orchestrator model is first and marked; list it first with "(Recommended)" (the picker cannot preselect). Extra ids the user types are allowed. Ask question 2a after question 2, as one single-select question per selected model (batch them, at most 4 per AskUserQuestion call). The recommended level is the one the user-level settings give that model today: its settingsEffort.perModel entry for the full model id, else settingsEffort.default. List it first with "(Recommended)". Fill the other options from effortLevels, at most 4 in all: on Claude leave max to "Other", unless max is the recommended level, in which case drop low instead. With no settings level (or on Junie) there is no recommendation; the CLI's built-in default is unknown, so the user picks. Always pin it, because an unpinned worker takes the observer's last /effort for that model, which changes between rounds without a trace (round 19). Ask question 3 only when the detected harness is Claude; Junie non-interactive has no bypassPermissions equivalent (--brave is interactive-only).

1. Instrumentation: server call log first (needs a plugin rebuild; the observer restarts MPS itself with mps_control.sh restart) or transcript-only?
2. Worker models (run list_worker_models.py; default: the orchestrator model from that list). 2a. Effort level for <model> (one per selected model; options from effortLevels; recommended: the level the user-level settings give that model today, if any). Every run of that model gets EFFORT=<level>.
3. Permission mode for workers (default: bypassPermissions on the developer's machine).
4. Scope of the first pass before gate 1 (default: S1 + S3 on the selected models). Gate 1 (after the pilot): matrix size. Gate 2 (after the report): which remedies; A/B yes/no.

Procedure (tick as you go; details in the references)

1. Preflight — MPS running with the MCP server enabled. The port belongs to the IDE selector (64343 on the 261 from-sources MPS, 64344 on 262); mps_control.sh and run_worker.sh detect it from the live launcher themselves (mps_control.sh url --json shows what they find). Do not export MPS_MCP_URL during preflight: MPS may not be up yet, and an exported unconfirmed value pins the whole round to the wrong port. Set it by hand only to override detection (e.g. several MPS processes), or after mps_control.sh wait has reported confirmed: true. SMOKE targets the harness project, never a developer checkout. Check the toolchain: claude --version (≥ 2.1; must accept --output-format stream-json --strict-mcp-config) when the detected harness is Claude, or junie --version when it is Junie, python3 -c 'import sys; assert sys.version_info >= (3, 9)', jq --version. Preflight is self-healing, and every step of it is yours:
   • MPS not running → mps_control.sh start (or, with no capture on file, the IDEA MPS run configuration), then mps_control.sh wait.
   • Welcome screen → synthesize a harness project and open it: python3 $STUDY/scripts/new_study_project.py --dir ~/MPSProjects/mcp-study/proj/harness writes the three descriptor files (migration.xml derived from this MPS, so no Migration Assistant), then open it via CLI (mps-project-management). Announce the path first. Do not retry MCP until that open has landed — Welcome-screen calls are rejected before dispatch.
   • mps_mcp_list_open_projects(projectPath=<harness>) must then list it. A synthesized project is empty by construction (mps_mcp_get_project_structure returns no modules); nothing in the study depends on a hand-maintained one, though an existing empty project may be substituted if synthesis fails. Every other project must be closed first — including the developer's own checkout, the common case when MPS was started from the IDEA run configuration. Disjoint module names are not enough: with confirmOpenNewProject2 = -1 (the default) the second open raises the modal New Window / This Window prompt and blocks the round (lesson 30). Announce the path you close; it is an observer action, not one to ask for.
   • Is the call log on? Ask the process, not the log: ps -ww -p $(pgrep -f '[j]etbrains\.mps\.Launcher') -o args= | tr ' ' '\n' | grep calllog must print the option, and the file must grow after a tool call. If it is off and gate question 1 said call-log, step 2 turns it on; if gate 1 said transcript-only, expect 0-line *-server.jsonl slices and skip the call-log checks below. Record the tool inventory: MPS_MCP_URL=$($STUDY/scripts/mps_control.sh url) python3 $STUDY/scripts/tools_inventory.py --out $RUNS/inventory.json (it does not detect the port itself). Run the harness's own unit tests once (cd $STUDY/scripts && python3 -m unittest discover -s tests -p 'test_*.py') — a broken script is cheaper to find here than in the evidence. Then run the contamination guard yourself: python3 $STUDY/scripts/check_user_agents.py (exit 0 clean, 3 contaminated). It rejects MPS-related Markdown definitions below ~/.claude/agents / ~/.junie/agents — a filename matching *mps* or a body containing mps_mcp, both case-insensitively — and any mps-* folder in ~/.claude/skills or ~/.junie/skills, which would shadow the per-project catalog and silently replace the thing being measured (lessons 26, 32). run_worker.sh runs the same guard before any run side effect. The guard never modifies anything: move an offending user skill out of the skills directory for the round and restore it at wrap-up. Built-in Explore and Task agents are outside this pin and remain enabled.
2. Instrument — the plugin logs one JSON line per dispatched call when MPS runs with -Dmps.mcp.calllog=<file> (McpCallLogListener, off by default). Turn it on without a human and without touching a tracked file: mps_control.sh capture while MPS is still alive, then mps_control.sh calllog $RUNS/server-calllog.jsonl (writes the option into the capture), then shutdown (the close of the last project carries the exit), start <project>, wait, and one SMOKE run as the readiness gate. capture only preserves VM options the live process already carries, which is why calllog exists — adding the option to the MPS run configuration works too but is study-only and must be reverted at wrap-up (lesson 13), so prefer the capture route. Confirm the relaunched process actually carries it (ps -ww -p <pid> -o args= | tr ' ' '\n' | grep calllog) and that the file grows.
3. Template — the empty fixture is synthesized, not snapshotted (scripts/new_study_project.py): three descriptor files, no doc surface possible, and a migration.xml derived from the MPS that will open it. Module-bearing fixtures (statechart, recipes*) are still tarballs, snapshotted without any agent doc surface: exclude .git, workspace.xml, and also .agents/, .claude/, AGENTS.md, CLAUDE.md. A tarball is a point-in-time copy, so a catalog inside it is what every later round measures no matter how far the bundled skills have moved (lesson 20). Instead, run_worker.sh installs the live catalog into each run's project right before launching the worker — scripts/install_skills.py purges every mps-* folder plus both guides and calls mps_mcp_initialize_project_for_agents, then records skillsSha256 in the meta. Verify every tarball: tar -tzf <f>.tar.gz | grep -E '(^|/)(\.claude|\.agents|AGENTS\.md|CLAUDE\.md)' must be empty (a synthesized project has nothing to verify). Do NOT put .mcp.json in the template; run_worker.sh generates the worker's MCP config per run into $RUNS/<id>-mcp/ from the detected URL and passes it with --strict-mcp-config.
4. Smoke — SMOKE is a harness check, not a scenario: a read-only prompt that lists open projects and stops, so it runs against the harness project itself (no template copy, no evaluation, pass stays empty). It is also the readiness gate after every MPS start or restart — a live process is not readiness. If the harness project is not open, announce its path, synthesize it if needed and open it via CLI; do not close it afterwards unless the next run needs a different project. RUNS=$RUNS EFFORT=<level for $MODEL> MAX_TURNS=6 PROJECT_SYNTHESIZED=1 $STUDY/scripts/run_worker.sh SMOKE $MODEL <n> <harness-project> — bump <n> on every re-run (the harness refuses an existing run id); drop PROJECT_SYNTHESIZED=1 if the harness project was not synthesized. The transcript must contain tool_use, tool_result, per-message usage; exactly one MCP server; and, when the call log is on, a SMOKE-…-server.jsonl slice of ≥ 1 line.
5. Scenarios — study/scenarios/S1..S10/{worker_prompt.md,done_criteria.md}. Which cells a changed skill actually forces is study/scenarios.md (brief list, then the directory). Look that up before picking the matrix; the gate-4 default (S1 + S3) is a pilot default, not that lookup. Add a scenario for whatever skill/tool the directory does not cover. S10 (project lifecycle) runs last in a round: it is the only scenario whose worker closes and opens projects, and a mistake in it can leave a modal dialog that blocks every later mps_mcp_* call. Prompts are developer-voice, fixed names, explicit "done", NO reporting requirements. Fixtures: empty-project (synthesized per run, not a tarball), statechart (Projectxx5), recipes (a passing S1) — regenerated per study/fixtures/README.md, not stored in git.
6. Runs — ONE scratch project open at a time (see lessons: shared module repository leaks across projects; S10 honours this by being sequential — it closes one project before opening the next). Restart MPS before a cell whose fixture language an earlier cell already loaded in the current process — any two of S3/S5/S6/S7/S9 on recipes*, S2/S8 on statechart (shutdown → start harness → wait → SMOKE; launch with ISOLATION=per-shared-fixture-restart, recorded per run beside mpsPid; lessons 40, 42). A read-only cell (S9) may precede a language-changing one in the same process; synthesized cells (S1, S10) need no restart. Per run: synthesize the empty project or copy the fixture tarball (PROJECT_SYNTHESIZED=1 when synthesized) → announce the scratch path (and any path you will close first) → close a previous scratch with mps_mcp_close_project if one is still open → open the new copy via CLI (mps-project-management) → confirm with list_open_projects → launch detached (run_worker.sh first rejects MPS-related user agents, then installs the live skills; either guard failure aborts with exit 3) → poll the PID in bounded loops → evaluate with an Opus subagent using the done_criteria.md (read-only mps_mcp_*, always with projectPath) → record pass/evidence in <id>.meta.json → announce the path and close with mps_mcp_close_project (force=false; on MODAL_BLOCKED ask the user only to dismiss the dialog). Sequential, never two workers against one MPS. Pass the model's gate-2a level as EFFORT on every launch. Check that every meta's effort is that level and that every meta's skillsSha256 and guidesSha256 are each the same value before comparing runs; a differing one means the catalog or the installed AGENTS.md / CLAUDE.md moved mid-round. Open/close details: references/harness.md.
7. Analyse — python3 $STUDY/scripts/analyze_runs.py $RUNS [--out DIR] (default $RUNS/analysis) → metrics.csv, tools.json, chains.json, errors.json, hotspots.md; pass is filled from each run's meta after evaluation. Then python3 $STUDY/scripts/families.py <baseline runs> <previous round> $RUNS > $RUNS/analysis/families.tsv (per-run family counters, references/analysis.md). Filter chains containing mps_mcp, group into families, have an Opus reviewer inspect 3 instances per family with study/scripts/show_steps.py and assign determinism {1.0, 0.5, 0}. Rank by avoidable turns (fixed context ≈ 150 K cache-read tokens per turn dominates) as well as by the study formula.
8. Classify each hotspot D → S → P-off → P-on → T (first fit). Write HOTSPOT_REPORT.md: baseline table, ranked hotspots with run:step evidence, hypotheses, defects, remedies with owner/contract/saving/risk. Keep a separate docs-defects.md from day one.
9. Treat — parallel Opus implementers with DISJOINT file sets (skill docs / skill scripts + packaging + drift test / server batch). Never two agents in the same Kotlin toolset; the observer registers new tests in McpToolsIntegrationTestSuite and runs the suite between batches. A server-side change needs the plugin rebuilt and MPS restarted: mps_control.sh restart + SMOKE, no human step.
10. A/B (optional) — restart MPS onto the treated plugin (mps_control.sh restart, then SMOKE), then the same runs against the treated tools; success = ≥ 30 % fewer tool calls and ≥ 25 % fewer context tokens on treated scenarios, no drop in pass rate; delete remedies that do not pay.
11. Wrap up — fold conclusions into the study doc; revert the VM option; delete fixture tarballs (keep the SMOKE scenario — step 4 needs it); announce and close any remaining scratch with mps_mcp_close_project, closing the last one with shutdownWithLastProject=true if MPS should go down — the shutdown rides on that last close, because a Welcome-screen MPS cannot be stopped over MCP; delete any <proj>-target directory an S10 run left; clean ~/MPSProjects/mcp-study/, ~/.claude.json project entries, and ~/.claude/projects/-…-mcp-study-proj-*/ memory dirs; Junie workers may leave ~/.junie/sessions — do not auto-delete them; keep the call-log listener. The capture file lives in $TMPDIR, outside that cleanup, so a later round can still relaunch.