Monitor Test Run

Overview

Given a testRunId, watch the run and narrate it. The skill:

1. Reads live_remediation_enabled once via getOrgSettings, so it can explain deterministically what happens when an execution is blocked.
2. Runs the bundled scripts/poll-test-run.js in the background — it watches the run and emits a line only when an execution's state changes (the model never hand-polls).
3. Reacts to those emitted lines: surfaces a blocker (with the live-remediation URL) when one appears, stays silent in between.
4. On the poller's DONE, does a post-mortem so a blocker is never mistaken for a pass.

The skill is conversational: its "output" is the messages it posts to the user (events + final summary), not a value returned to a caller. It changes nothing server-side except, optionally, terminateTestRun if the user asks to stop the underlying run.

Tool naming. This doc refers to Kobiton MCP tools by their bare names (getOrgSettings, terminateTestRun). The registered name depends on how the host loaded the MCP server (Claude Code as a plugin exposes mcp__plugin_automate_kobiton__getOrgSettings; a repo-local .mcp.json exposes mcp__kobiton__getOrgSettings; other hosts differ). Use the bare name and let the host resolve the prefix. (Run state is read by the bundled poller over REST, not via the MCP getTestRun tool — a background process can't call MCP tools.)

Prerequisites

Runs on any OS, but needs a local filesystem, Node.js 18+, and ~/.kobiton/.credentials — the watch is a bundled Node poller that reads that file. Where you can't run a local command at all, tell the user to follow the run in the portal instead. A streamed-output affordance is strongly preferred but not required: Claude Code uses the Monitor tool, other hosts substitute their own streamed shell / watch / loop, and a host with none falls back to the foreground loop in Step 2 — see that per-host table, including what NOT to use (a silent detached process misses the blocker window). See the Skill compatibility matrix in CLAUDE.md.

• ~/.kobiton/.credentials — scripts/poll-test-run.js reads it directly on each invocation and has no MCP fallback, so this file is required. If it's missing, say so and point at /automate:setup rather than starting a poller that will fail on its first REST call.
• An authenticated Kobiton MCP connection. All three tools resolve the caller's org/user from the OAuth context.
• A testRunId — usually the one createTestRun just returned, or one the user names.
• getOrgSettings available. If the host can't find it, fall back to the flag-OFF branch (see Step 1) rather than failing the watch.

Inputs

Status vocabulary (read before Step 3)

getTestRun returns revisit_executions[], each with: { id, test_case_id, status, failure_type, execution_session_id, assigned_device_id, execution_error }. assigned_device_id is the device the execution runs on — use it directly to build the live-remediation URL (Step 4); no separate session lookup is needed.

Execution status values: NEW, SCHEDULED, RUNNING, UPLOADING_IMAGE, COMPLETED, TERMINATING, NETWORK_PAYLOAD_SCANNING. COMPLETED is the terminal status; what kind of terminal it is comes from failure_type.

failure_type values (the ones this skill keys on): NONE (clean), BLOCKER_ENCOUNTERED, TERMINATED_BY_USER, TERMINATED_BY_SYSTEM, and several genuine failures (NETWORK_ISSUE, INIT_SESSION_FAILED, APP_CRASHED, TARGET_ELEMENT_NOT_FOUND, OTP_CODE_ENCOUNTERED, …).

The blocked moment depends on the flag — this is the key behavior:

• Live remediation ON: when an execution hits a blocker it pauses in a blocked-waiting state (live status BLOCKED_WAITING, then BLOCKED_RESUMING once remediation is accepted) and waits for the user to resolve it interactively. It does not end yet.
• Live remediation OFF: there is no waiting state. The execution ends immediately — status = COMPLETED, failure_type = BLOCKER_ENCOUNTERED — and the user's later portal resolution applies on the next rerun.

So with the flag OFF you will typically never observe a blocked-waiting status: the first you see of the blocker is the terminal COMPLETED + BLOCKER_ENCOUNTERED. With the flag ON you see the pause first, then either a resume or (if the wait expires / is not resolved) a terminal COMPLETED + BLOCKER_ENCOUNTERED.

Steps

1. Set up once: flag, portal base, and (if flag ON) the open-on-blocker preference

Do all of this before the watch loop, so when a blocker hits later there is nothing left to decide.

1a. Read the live-remediation flag. Call getOrgSettings a single time and cache settings.live_remediation_enabled as flagOn for the rest of the run — do not re-read it on every poll. Fallback (safer default): if getOrgSettings errors or has no live_remediation_enabled, treat flagOn = false and tell the user once that you couldn't read the flag and are assuming OFF.

1b. Derive the portal base URL from the MCP server the run lives on — do not hardcode https://portal.kobiton.com. Read the MCP server URL from .mcp.json and map the api host to its portal equivalent (drop any trailing /mcp), same rule as run-automation-suite:

So a run watched through api-test.kobiton.com → https://portal-test.kobiton.com, api-test-green → portal-test-green, etc. Cache this as <portal>. If the mapping can't be resolved, fall back to https://portal.kobiton.com. Use the MCP server that serves getTestRun for this run — i.e. the env the run and its devices live on — not the env any cross-org flag read happened to use.

1c. State the flag state, and — if flagOn = true — settle the open-on-blocker preference up front.

• flagOn = false → "Live remediation is not enabled for your org — a blocked execution ends immediately with BLOCKER_ENCOUNTERED, and a resolution you submit in the portal applies on the next rerun." (Nothing to pre-arrange; there is no live window to open. autoOpen is irrelevant.)
• flagOn = true:
  • If autoOpen was already decided by the caller (e.g. the create-test-run skill delegated here after the user chose "monitor + auto-open" → autoOpen = yes, or "monitor only" → autoOpen = no), do not ask again — just state it: "Live remediation is enabled; I'll {auto-open the live-remediation window on a blocker / surface the URL on a blocker}." Skip straight to Step 2.
  • Otherwise (invoked directly, no pre-set preference) → "Live remediation is enabled — if an execution hits a blocker it pauses and waits for interactive remediation." Then ask now, once, before monitoring: "When a blocker hits, do you want me to automatically open the live-remediation window for the blocked device?" Cache the answer as autoOpen (yes/no). Asked a single time up front — not per blocker mid-run.

2. Stream the bundled poller — do NOT hand-poll

Do not write your own sleep/poll loop and do not narrate individual polls. That is exactly what made a prior run noisy — calling getTestRun over and over and posting "still NEW / still blocked, no change". The bundled poller emits a line only when an execution's state actually changes and exits when the run is done:

node $SKILL_DIR/scripts/poll-test-run.js --run-id <testRunId>

($SKILL_DIR is this skill's directory. Optional: --interval <sec> base cadence (default 3), --max-interval <sec> backoff cap (default 30), --max-errors <n> consecutive-error give-up (default 5), --waiting-heartbeat <sec> blocked-heartbeat cadence (default 60, 0 disables).) The script reads run state via REST using ~/.kobiton/.credentials (a background process can't call the MCP getTestRun tool — that's why this is a script). It diffs per execution, backs off during quiet stretches, and exits 0 on DONE.

Run it so each emitted line streams back to you as it happens — this is the whole point. A plain "background" launch is NOT enough: a backgrounded process's stdout does not re-engage you, so you'd sit idle and miss the blocker window (this was a real failure). Use the host's stream-a-command's-output mechanism so every line re-invokes you:

Whichever mechanism: you react only to the lines it prints. Stay silent between lines; there is no value in reporting "no change". The lines:

Short-circuit: if the run is already fully terminal, the script emits the terminal EVENTs then DONE on the first poll — same path, no special-casing needed.

The event kinds map to outcomes as follows (the script applies this; it's here so you narrate correctly): terminal_passed = COMPLETED + failure_type NONE; terminal_blocker_encountered = COMPLETED + BLOCKER_ENCOUNTERED (a blocker, never a pass); terminal_terminated = COMPLETED + TERMINATED_BY_USER/TERMINATED_BY_SYSTEM; terminal_failed = COMPLETED + any other failure_type.

4. Surface blockers (loud); stay quiet on passes

On an EVENT blocked line (flag ON, paused) or an EVENT terminal_blocker_encountered line (flag OFF, already ended) from the poller — post a message containing:

• Which execution (exec=) and, for a suite, which test case (tc=) is blocked.

• The device — the device= field on the event line (the poller reads it from assigned_device_id).

• The live-remediation URL — built from <portal> (the base from the poller's READY line, env-mapped, not hardcoded to production) and device=. Note: no &view=device-only — the full default device-launch view is wanted here so the Kobi AI chat panel and controls are visible alongside the device:

  <portal>/devices/launch?id=<device>

• A deterministic explainer keyed on the cached flagOn:

  • flagOn = true → this is an action request, not a status update — the execution is now waiting on the user, on a clock. Say so plainly and urgently, e.g.: "⏳ Action needed now — this execution is paused on a blocker and waiting for you. It will auto-fail with BLOCKER_ENCOUNTERED if the remediation window times out, so resolve it promptly. {If autoOpen = yes: "I've opened the live-remediation window —" else: "Open the URL to"} chat with Kobi AI / drive the device to clear the blocker; the execution resumes automatically once it's cleared." Do not frame this as "I'll just wait and ping you when something changes" — by the time the next change arrives it may already be the timeout.
  • flagOn = false → "Live remediation isn't enabled for your org, so this execution ended with BLOCKER_ENCOUNTERED. Submit your resolution in the portal — it applies on the next rerun."

If the event line shows device=- (the device id is unexpectedly absent — it is part of the getTestRun contract, so this points to a backend/state problem): surface the blocker text without a URL and note INVALID_EXECUTION_STATE (missing assigned_device_id). Don't fabricate a device id.