Offload

Running Tests with Offload

Offload is a parallel test runner that distributes test execution across sandboxes (local processes or remote Modal environments). This skill covers invoking tests, reading results, and debugging failures.

Installation

If the offload binary is not on PATH, install it:

cargo install offload

How to Invoke Tests

Use the first approach that applies:

1. Look for existing invocation commands

Check Makefile, justfile, Taskfile, package.json scripts, and scripts/ for targets that wrap offload run. Prefer these -- they encode project-specific flags (copy-dirs, env vars, config paths).

# Examples of what to look for:
just test                       # justfile target
make test-offload               # Makefile target
./scripts/offload-tests.sh      # shell wrapper

2. Use offload run directly

If no wrapper exists, invoke Offload directly from the project root (where offload.toml lives):

offload run                                 # basic run
offload run --parallel 8                    # override parallelism
offload run --copy-dir ".:/app"             # copy cwd into sandbox at /app
offload run --env KEY=VALUE                 # set sandbox env var (repeatable)
offload run --no-cache                      # force fresh image build
offload run --collect-only                  # discover tests without running
offload run --show-estimated-cost           # show sandbox cost after run
offload run -c path/to/offload.toml         # use alternate config

3. Fall back to non-Offload commands

If Offload is not installed or Modal credentials are unavailable, use the project's native test command (e.g. cargo nextest run, pytest).

When to Use Offload

Use Offload when:

• Running integration or end-to-end test suites
• Total test runtime exceeds ~2 minutes
• Multiple agents are working concurrently and competing for local CPU
• The project already has an offload.toml

Skip Offload when:

• Running a single test during TDD iteration (use the native runner directly)
• Tests require local-only resources (hardware devices, localhost services not reachable from sandboxes)
• No offload.toml exists and the task does not call for setting one up

Exit Codes

Debugging Failed Tests

Run summary

After a run completes, offload prints a summary:

Test Results:
  Total:   128
  Passed:  126
  Failed:  2
  Duration: 6.01s
  Estimated cost: $0.0004 (11.1 CPU-seconds)

The Estimated cost line appears when --show-estimated-cost is passed to offload run. Use the summary to confirm tests ran and gauge the scope of failures before diving into logs.

Reading logs

Important: If you ran with -c path/to/config.toml, pass the same -c flag to offload logs. Logs are stored in the config's output_dir, so mismatched configs will show stale or missing results.

Always filter offload logs output to avoid flooding your context window. Never run bare offload logs on a large suite. Follow this workflow:

1. Check the run summary to see how many tests failed.
2. Retrieve failure output (choose based on what fits your context window):
   • Use --failures to see all failures at once.

     offload logs --failures

   • Use --test or --test-regex to isolate a specific test.

     offload logs --test "path/to/test.py::test_name"  # exact test ID
     offload logs --test-regex "test_math"             # regex substring match

   Filters combine with AND logic:

   offload logs --failures --test-regex "database"

3. Fix and rerun.

Each test is separated by a banner showing its ID and status. The test ID format varies by framework:

=== tests/test_math.py::test_div [FAILED] ===
AssertionError: expected 2 got 3

=== trace::tests::test_active_tracer [FAILED] ===
assertion `left == right` failed
  left: 2
 right: 3

Flaky tests

If a test fails intermittently, add or adjust a group with retries in offload.toml:

[groups.flaky]
retry_count = 3
filters = "-k test_flaky_name"

Run offload validate after editing to check config syntax. A test that fails then passes on retry exits with code 2 (flaky).

Common failure patterns

Image Cache

Offload caches image IDs in git notes (refs/notes/offload-images). Notes are keyed by commit SHA and TOML config path, so multiple configs in the same repo do not collide. Notes are fetched from and pushed to the remote automatically. Pass --no-cache to offload run to skip reading and writing the cache; --no-cache preserves the same build procedure (tree export, base image build, thin diff) -- it only suppresses note interactions.

Image Caching Modes

Offload uses a unified pipeline for image caching. Both modes follow identical steps after resolving the base commit: cache lookup, tree export, base image build, thin diff application, and note write. The only difference is how the base commit is selected.

Latest-commit mode (default)

When no [checkpoint] section is present, Offload uses the latest commit (HEAD) as the base:

1. Look up HEAD in git notes for a cached base image.
2. Cache hit: generate a binary diff from HEAD to the working tree and apply it on top of the cached image.
3. Cache miss: export the HEAD tree, build a base image from it, cache the result in git notes on HEAD, then apply thin diff.
4. Empty repo (no commits): fall through to a full build, no caching.