Runtime Observation

Runtime Observation Methodology

Runtime observation produces ground truth. When you execute a command and record the output, that observation is an empirical fact -- not an interpretation, not an inference, not a guess. This skill defines how every runtime observation agent operates.

The Principle

Run it. Record it. Cite it.

You never guess what a target does. You run it and document what happens. Every observation is a witnessed fact: this input produced this output in this environment at this time.

When Runtime Observation Applies

Runtime observation applies to any target that can be executed inside a container:

Targets that cannot be executed (static documentation, binary files without a runtime) are handled by other intelligence sources. Runtime observation is additive -- it enhances and corroborates intelligence from all other modes.

Source Origin

All runtime observation output is RAW. Running the target and recording its behavior produces artifacts that require sanitization before reaching the implementer. Even though the observations describe external behavior (not implementation internals), recording the target's responses creates a derivation chain.

Targets that cannot be executed in a sandbox (or where executing them has side effects the user declines to accept) should skip this mode.

• Output location: workspace/raw/runtime/
• Never write runtime observations to workspace/output/ or workspace/public/.
• The origin label does not reduce the value of the observations. It is a provenance marker that controls downstream processing.

Container Requirement

All runtime observation happens inside containers. No exceptions. The target is never executed on the host machine. This is a hard safety requirement. The container is already running before any runtime observation agent begins work -- the orchestrator sets it up per the container-execution skill.

Agents MUST NOT attempt to build, start, stop, or remove containers. They interact with an already-running container.

Five-Phase Observation Methodology

Every runtime observation agent follows these five phases in order. You are not required to complete every phase for every target -- proceed as far as the target type and available time allow. But you MUST follow the phase order.

digraph observation_phases {
    rankdir=TB;

    "Start runtime observation" [shape=doublecircle];
    "Phase 1: Explore target surface area" [shape=box];
    "Phase 2: Test nominal paths with valid inputs" [shape=box];
    "Phase 3: Probe boundaries and limits" [shape=box];
    "Phase 4: Trigger error conditions" [shape=box];
    "Phase 5: Explore state transitions" [shape=box];
    "Convert observations to behavioral claims" [shape=box];
    "Observation complete" [shape=doublecircle];

    "Start runtime observation" -> "Phase 1: Explore target surface area";
    "Phase 1: Explore target surface area" -> "Phase 2: Test nominal paths with valid inputs";
    "Phase 2: Test nominal paths with valid inputs" -> "Phase 3: Probe boundaries and limits";
    "Phase 3: Probe boundaries and limits" -> "Phase 4: Trigger error conditions";
    "Phase 4: Trigger error conditions" -> "Phase 5: Explore state transitions";
    "Phase 5: Explore state transitions" -> "Convert observations to behavioral claims";
    "Convert observations to behavioral claims" -> "Observation complete";
}

Phase 1: Exploration

Goal: Discover what the target can do.

Activities:

• Run help commands (--help, -h, help, man target)
• Read menus, navigation bars, landing pages
• List available API endpoints (OpenAPI, GraphQL introspection, sitemap)
• Discover configuration options (environment variables, config files, CLI flags)
• Identify authentication requirements (does the target prompt for credentials?)
• Check version information (--version, -V, /api/version)

Output: A map of the target's surface area. What commands exist. What endpoints respond. What configuration is available. This is the foundation for all subsequent phases.

Provenance: Exploration observations are confidence=confirmed (the target responded with this output). Inferences about what a discovered feature does are confidence=inferred until tested.

Phase 2: Nominal Path Testing

Goal: Exercise the target's happy paths -- standard workflows with valid inputs that produce expected outputs.

Activities:

• Execute each command with the simplest valid arguments
• Submit forms with valid data
• Make API requests with well-formed payloads
• Follow the "getting started" workflow end-to-end
• Perform CRUD operations where applicable (create, read, update, delete)
• Test each output format option (JSON, text, CSV, etc.)

Recording discipline: For each nominal test, record:

• The exact input (command, request, form data)
• The exact output (stdout, response body, UI state)
• The exit code or HTTP status
• Any side effects (files created, state changed)

Provenance: Nominal path observations are confidence=confirmed. Each recorded input/output pair is a reproducible fact.

Phase 3: Boundary Testing

Goal: Probe the edges of the target's input domain to discover validation rules, limits, and type handling.

Activities:

• Empty inputs: What happens with zero-length strings, empty files, no arguments?
• Maximum sizes: What is the largest input accepted? When does truncation occur?
• Special characters: Unicode, null bytes, newlines in arguments, shell metacharacters
• Type boundaries: Negative numbers, zero, MAX_INT, floating point precision
• Format variations: Dates in different formats, URLs with unusual schemes, paths with spaces
• Missing required fields: Omit each required field one at a time

Recording discipline: For each boundary test, record whether the target:

• Rejected the input with a clear error message
• Accepted the input and processed it
• Crashed or hung
• Produced unexpected output

Provenance: Boundary observations are confidence=confirmed. The target's validation behavior (or lack thereof) at each boundary is an empirical fact.

Phase 4: Error Probing

Goal: Deliberately trigger error conditions to document the target's error handling, error messages, and failure modes.

Activities:

• Authentication failures: Wrong credentials, expired tokens, missing auth headers
• Permission failures: Read-only paths, operations requiring elevation
• Resource failures: Non-existent files, unreachable hosts, full disks (simulated)
• Timeout behaviors: Commands that take longer than the timeout
• Concurrent access: Multiple simultaneous requests (where applicable)
• Invalid state transitions: Operations in the wrong order, double-close, resume after finish
• Malformed input: Invalid JSON, truncated payloads, wrong content types

Recording discipline: For each error probe, record:

• The exact trigger (what caused the error)
• The error message (exact text)
• The error code (exit code, HTTP status, error type)
• Recovery behavior (does the target recover or remain in a broken state?)

Provenance: Error probing observations are confidence=confirmed. Error messages and codes are empirical facts. Interpretations of what the error means (e.g., "this exit code indicates a permission problem") are confidence=inferred.

Phase 5: State Exploration

Goal: Understand how the target maintains and transitions between states.

Activities:

• Session lifecycle: Create, use, suspend, resume, expire, destroy sessions
• Configuration persistence: Change a setting, restart the target, verify the setting persists
• Data persistence: Create data, restart the container, verify data survives
• Cache behavior: First request vs. second request timing differences
• Initialization sequence: What happens on first run vs. subsequent runs?
• Cleanup behavior: What does the target do on graceful shutdown vs. kill?

Recording discipline: State exploration requires before/after snapshots:

# Before: capture state
$RUNTIME exec $CONTAINER sh -c 'find /app /tmp -type f -newer /tmp/start-marker 2>/dev/null' \
  > workspace/raw/runtime/observations/state-before.txt

# Action: perform the state change
$RUNTIME exec $CONTAINER sh -c 'target create --name test 2>&1'

# After: capture new state
$RUNTIME exec $CONTAINER sh -c 'find /app /tmp -type f -newer /tmp/start-marker 2>/dev/null' \
  > workspace/raw/runtime/observations/state-after.txt

# Diff: what changed
diff workspace/raw/runtime/observations/state-before.txt \
     workspace/raw/runtime/observations/state-after.txt \
  > workspace/raw/runtime/observations/state-diff.txt

Provenance: Directly observed state transitions are confidence=confirmed. Inferences about internal state management (e.g., "the target probably uses a file-based session store because session files appear in /tmp") are confidence=inferred.