Examples QA

Instructor examples QA

Use the examples suite as an end-to-end check of Instructor behavior, but keep the execution mode explicit. A live run tests provider and transport integration; a record run creates reusable HTTP fixtures; a replay run tests the current code against fixed provider responses without spending tokens or contacting a network.

Inspect before running

Run commands from the repository root. Discover the target and its canonical selector first:

php bin/instructor-hub list
php bin/instructor-hub show getters_and_setters

The hub accepts a front-matter docname, numeric index, short ID such as xe4bb, or example path. Read the target's run.php before diagnosing it. Check its tags and whether it builds an HTTP client implicitly or supplies a custom client/mock.

Treat these sources as authoritative when the prose documentation differs:

• just/examples.just — preferred record/replay command wrappers
• examples/boot.php and examples/_support/HttpRecordingBoot.php — mode and recording-directory behavior
• packages/hub/src/Commands/EnhancedRunAllExamples.php — corpus filtering and OK/ASSERT/ERROR/FLAKY classification
• packages/hub/src/Services/ExampleRepository.php — selector resolution
• QUALITY.md — repository QA policy and operational caveats

Choose the execution mode

Use the smallest mode that answers the question:

Run without record/replay

Use pass mode for a normal baseline or a live provider smoke check:

INSTRUCTOR_EXAMPLES_HTTP=pass \
  php bin/instructor-hub run getters_and_setters

Omitting INSTRUCTOR_EXAMPLES_HTTP has the same effect because pass is the default. Pass mode attaches no ambient middleware. LLM examples therefore need their normal provider keys and network access; mock-only examples remain local.

For a code change, prefer this sequence:

1. Run a narrow live or existing-fixture baseline when provider integration is in scope.
2. Record only the affected examples if the fixture is missing or intentionally refreshed.
3. Replay the same selector to verify deterministic behavior.
4. Run the replay gate for the affected group, then the full replay corpus when practical.
5. Run package tests separately when the example result does not identify the failing layer.

Do not treat a replay pass as proof that provider authentication, DNS, TLS, rate limits, or current provider behavior work. Do not treat a live LLM failure as a product regression until it has been reproduced with replay or an appropriate unit/integration test.

Record one example

The preferred scratch-fixture workflow is:

just examples-record getters_and_setters

The wrapper sets:

INSTRUCTOR_EXAMPLES_HTTP=record
INSTRUCTOR_EXAMPLES_RECORDINGS_DIR=examples-recordings

The live response is stored under a namespaced directory such as:

examples-recordings/A01_Basics/BasicGetSet/
├── cassette.json
└── interactions/000001/
    ├── interaction.json
    └── response.body                 # or response.chunks.ndjson

Record the whole corpus only deliberately; it requires keys for every provider used and can be expensive:

just examples-record-all --limit=20
just examples-record-all --filter=errors

To create fixtures in each example's co-located recordings/ directory, omit the recordings-directory override:

INSTRUCTOR_EXAMPLES_HTTP=record \
  php bin/instructor-hub run getters_and_setters

The direct hub sugar is equivalent and is useful when keeping the mode visible at the invocation site:

php bin/instructor-hub run getters_and_setters --record
php bin/instructor-hub all --record --limit=20

The bin/instructor-hub launcher consumes --record, --replay, and --recordings-dir=<dir> before dispatching the hub command. If using a custom directory, use the same directory for record and replay:

php bin/instructor-hub run getters_and_setters \
  --record --recordings-dir=examples-recordings

Recording is live. Confirm the relevant provider key exists, expect token/network cost, and do not record broad personal or production data. Common credentials are sanitized before persistence, but prompts, model outputs, PII, and provider-specific secrets still require review.

Streaming recordings are published only after the stream completes naturally. An interrupted or failed stream must not be treated as a valid fixture.

Replay selected examples

Replay from the same root used for recording:

just examples-replay getters_and_setters

Or use the direct form:

php bin/instructor-hub run getters_and_setters --replay \
  --recordings-dir=examples-recordings

For keyless replay through a raw example path:

INSTRUCTOR_EXAMPLES_HTTP=replay \
INSTRUCTOR_EXAMPLES_RECORDINGS_DIR=examples-recordings \
php examples/A01_Basics/BasicGetSet/run.php

Replay is hermetic by default for requests that enter the configured ambient middleware. It is not a blanket network sandbox: examples that construct their own client can bypass the hook and need the no-replay treatment or separate inspection. For intercepted requests, the examples boot hook provisions dummy provider keys only so configuration resolution succeeds; the recorded response should be served before any provider driver can authenticate or open a network connection. A missing recording, request mismatch, exhausted ordered session, corrupt payload, or unsupported cassette version is a diagnostic failure, not a reason to fall back to the live provider.

Replay the corpus with optional controls:

just examples-replay-all --limit=20
just examples-replay-all --filter=errors
php bin/instructor-hub all --replay --dry-run
php bin/instructor-hub all --replay --filter=errors

--dry-run verifies the selected set without executing it. Use --filter=errors after a tracked run to focus on previously failing examples; use --force when the hub's status cache would otherwise omit an example.

Understand what is actually covered

The ambient hook in examples/boot.php attaches middleware through HttpClientDefaults only when the example's runtime resolves an implicit HTTP client. Examples that construct their own client, use a mock driver, or are marked no-replay may not exercise the cassette path. Inspect the example before calling an unchanged result evidence for record/replay.

Corpus filtering is intentional:

• skip: true examples are never selected.
• The broken tag skips known-broken examples in both modes.
• The no-replay tag skips examples that cannot be recorded/replayed when replay mode is active; the hub logs the skipped set.
• The flaky tag describes live LLM nondeterminism. It does not make a replay failure acceptable.

No individual example needs to be edited to enable the shared mode switch. Avoid bulk-editing the example corpus just to add record/replay setup.