Offload Onboard

Onboard Repository to Offload

This skill walks through onboarding the current repository to use Offload — a parallel test runner that executes tests across Modal cloud sandboxes. Offload is installed as part of the procedure below.

Procedure

Follow these steps in order. Use the TodoWrite tool to track progress through each step.

Step 1: Detect Test Framework and Test Paths

Investigate how the repository runs its tests:

1. Look for pyproject.toml, setup.cfg, setup.py, Cargo.toml, package.json, go.mod, or similar project files
2. Look for existing CI workflows (.github/workflows/, .gitlab-ci.yml, etc.) to see how tests are currently invoked
3. Look for test directories: tests/, test/, **/test_*.py, **/*_test.go, src/**/tests/, etc.
4. Determine:
   • Framework: pytest, cargo, vitest, or default (generic)
   • Test paths: Where tests live (e.g. ["tests/"], ["src/"])
   • Python runner: If pytest, determine if the project uses uv, poetry, pip, or plain python
   • Extra args: Any special invocation needed (e.g. ["run", "--group", "test"] for uv with dependency groups)

Ask the user to confirm your detection if anything is ambiguous.

Step 2: Verify Prerequisites

Verify the following are installed and authenticated. Do not continue until all prerequisites are confirmed.

• uv (required — Offload uses uv to run the Modal sandbox script regardless of project language or package manager)
• modal CLI — must be installed (pip install modal) and authenticated. Run modal profile list to check. If not authenticated, tell the user to run modal token new (opens a browser, writes credentials to ~/.modal.toml). Wait for the user to confirm authentication before proceeding.
• rsync (required — Offload uses rsync for working directory snapshots during the full-build fallback path. Without it, if the thin-diff path fails, the entire job fails with no recovery)
• For pytest projects: the configured Python runner (uv, poetry, or python) and pytest must be available locally for test discovery
• For cargo projects: cargo-nextest must be installed (cargo install cargo-nextest)

Step 3: Find or Create a Dockerfile

Offload's Modal provider needs a Dockerfile to build sandbox images. Look for an existing one:

1. Check .devcontainer/Dockerfile, Dockerfile, docker/Dockerfile, or any Dockerfile referenced in CI
2. If one exists and is suitable (has the language runtime + package manager), note its path
3. If none exists, create .devcontainer/Dockerfile with the minimal base:

For Python projects:

FROM python:<version>-slim

RUN apt-get update && apt-get install -y --no-install-recommends rsync \
    && rm -rf /var/lib/apt/lists/*

# Only install git if tests or dependencies require it (not needed for thin diffs)
# RUN apt-get update && apt-get install -y --no-install-recommends git \
#     && rm -rf /var/lib/apt/lists/*

# Install the package manager used by the project (uv, poetry, etc.)
# For uv:
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
# For poetry:
# RUN pip install poetry

WORKDIR /app

For Rust projects:

FROM rust:<version>-slim

# Only install git if tests or dependencies require it (not needed for thin diffs)
RUN apt-get update && apt-get install -y --no-install-recommends pkg-config libssl-dev rsync \
    && rm -rf /var/lib/apt/lists/*

# Install cargo-nextest if the project uses it
# RUN cargo install cargo-nextest

WORKDIR /app

Key principles:

• Match the language/runtime version from the project's config (e.g. requires-python, rust-version)
• Include git if any tests create git repos or the project uses git-based dependencies
• Do NOT COPY . . — Offload overlays source via --copy-dir at image build time
• Keep it minimal — dependencies are installed at runtime inside the sandbox

Step 4: Create offload.toml

Create offload.toml at the project root using the Modal provider. In the templates below, values in <angle-brackets> are placeholders you must substitute with project-specific values.

[offload]
max_parallel = 3
test_timeout_secs = 120
sandbox_repo_root = "/app"

[provider]
type = "modal"
dockerfile = "<path-to-dockerfile>"
include_cwd = true

[framework]
type = "pytest"
paths = ["<test-paths>"]
command = "<pytest-command>"      # e.g. "uv run pytest", "poetry run pytest", "python -m pytest"

[groups.all]
retry_count = 0
filters = ""                    # pytest args for discovery filtering (e.g. "-m 'not slow'")

[report]
output_dir = "test-results"
junit = true
junit_file = "junit.xml"

For Cargo (Rust) projects, replace the [framework] section with:

[framework]
type = "cargo"

Optional: sandbox_init_cmd for build-time setup

Do not add sandbox_init_cmd initially. If you observe that every sandbox runs the same setup work (e.g. uv sync --all-packages, git apply, database migrations), move that work into sandbox_init_cmd so it runs once during image build instead of on every sandbox:

[offload]
max_parallel = 3
test_timeout_secs = 120
sandbox_repo_root = "/app"
sandbox_init_cmd = "uv sync --all-packages"   # runs once at image build time

This is especially useful for monorepo setups where dependency installation is slow.

Optional: post_patch_cmd for derived artifact regeneration

If the project generates code or builds assets that must stay in sync with source (e.g., API clients from OpenAPI schemas, frontend bundles, compiled protobufs), add post_patch_cmd to regenerate them after the thin-diff patch is applied:

[offload]
post_patch_cmd = "make generate-client"

Unlike sandbox_init_cmd (which runs once during base image builds), post_patch_cmd runs after every patch, ensuring derived artifacts match the patched source. The OFFLOAD_PATCH_FILE env var is set when a diff exists.

Optional: skip coverage tracing during discovery (pytest-cov)

If the project uses pytest-cov, coverage tracing runs even during pytest --collect-only, roughly doubling local discovery time. Set discovery_args under [framework] so Offload skips it during collection (discovery only — test-run coverage is unaffected):

[framework]
type = "pytest"
command = "uv run pytest"
discovery_args = "--no-cov"

Only set this when pytest-cov is installed; pytest --collect-only --no-cov is a usage error without the plugin. Keep it out of group filters, which are test selectors.

When to use type = "default" for the framework:

The built-in pytest and cargo frameworks cover straightforward setups. Fall back to type = "default" for the [framework] section when:

• Conflicting local config: The project's pyproject.toml or setup.cfg has addopts that conflict with Offload (e.g. xdist workers, coverage plugins) and you need to override them with -o addopts= or -p no:xdist
• Custom discovery pipeline: Standard collection doesn't work and you need shell pipelines (e.g. grep filtering, marker exclusions combined with workspace sync)
• Unsupported framework: Jest, Go, Mocha, or any framework not directly supported

Example — pytest in a monorepo with xdist conflict (still using the Modal provider):

[provider]
type = "modal"
dockerfile = "<path-to-dockerfile>"
include_cwd = true

[framework]
type = "default"
discover_command = "uv run pytest --collect-only -q {filters} 2>/dev/null | grep '::'"
run_command = "cd /app && uv run pytest -v --tb=short --no-cov -p no:xdist -o addopts= --junitxml={result_file} {tests}"
test_id_format = "{name}"

In monorepo setups where tests run from a subdirectory, add sandbox_project_root to set the test working directory separately from the repo root:

[offload]
sandbox_repo_root = "/app"
sandbox_project_root = "/app/mypackage"

[framework]
type = "pytest"
command = "uv run pytest"

Note: if discovery or execution requires pre-steps like uv sync --all-packages, use sandbox_init_cmd in the [offload] section rather than inlining them into the discover/run commands.

For the full configuration reference and more examples, see the Offload README.

Configuration reference for fields used above:

[offload]

• max_parallel: Number of concurrent Modal sandboxes (start with 3, optimize later)
• test_timeout_secs: Per-test-batch timeout in seconds (120s is generous for unit tests)
• sandbox_repo_root: Path to the repository root inside the sandbox (e.g. /app). Used for thin-diff patches and as the default test working directory (OFFLOAD_ROOT)
• sandbox_project_root: (optional) Working directory for test execution, if different from sandbox_repo_root. Only needed in monorepo setups where tests run from a subdirectory
• sandbox_init_cmd: Optional command to run during image build, after cwd/copy-dirs are applied (e.g. "uv sync --all-packages")
• post_patch_cmd: Optional command to run after thin-diff patch is applied, before image materialization (e.g. "make generate-client")

[provider] (Modal)

• dockerfile: Path to the Dockerfile for building the sandbox image
• include_cwd: Copy the current working directory into the image (default: false)

[framework] (pytest)

• paths: Directories to search for tests (default: ["tests"])
• command: Full command prefix for pytest invocation (e.g. "uv run pytest"). Replaces the legacy python/extra_args fields
• discovery_args: Extra args appended only during discovery (--collect-only), not execution — e.g. --no-cov to skip coverage tracing when the project uses pytest-cov

[groups.<name>]

• retry_count: Number of retries for failed tests (0 = no retries, 1 = catches transient failures)
• filters: Filter string passed to the framework during discovery (e.g. -m 'not slow')

IMPORTANT: Do not use JUnit reporting plugins or hooks. Offload owns the JUnit XML report — it passes --junitxml to pytest and uses the resulting XML as the single source of truth for matching test results back to discovered tests. Custom conftest hooks, plugins, or pytest options that modify JUnit XML output (e.g. record_xml_attribute, pytest-metadata, or junit_family overrides) will interfere with Offload's test ID resolution and cause incorrect results. If the project has existing JUnit customization in conftest.py or pytest plugins, remove it before proceeding.

Step 5: Verify Vitest Test Discovery (vitest only)

Skip this step if the framework is not vitest.

Offload's vitest integration uses --testNamePattern to select specific tests for parallel execution. This flag matches tests by their space-separated describe/test chain — not by file path. When two tests in different files share the same space-separated name, Offload cannot distinguish them and will error during discovery, blocking all test execution.

Run offload collect to verify that test discovery works:

offload collect

If Offload reports "Duplicate test names found (space-separated)", stop and present the duplicates to the user. Explain:

Offload cannot run vitest tests that share the same space-separated name across different files. This is a requirement of Offload's --testNamePattern-based test selection — without unique names, Offload will error during test discovery and no tests will run.

Then ask: "Would you like me to deduplicate these test names by making them more descriptive? This typically involves wrapping tests in uniquely named describe() blocks or renaming test()/it() calls to include more context."

If the user agrees, rename the tests to make the space-separated names unique. Prefer wrapping in descriptive describe() blocks over renaming individual tests, as this preserves the existing test names while adding disambiguation.

After making changes, run offload collect again. Repeat until offload collect succeeds with no duplicate name errors.

If the user declines deduplication, do not proceed with Offload onboarding — inform them that Offload's vitest integration cannot function with duplicate test names and they will need to resolve this before onboarding can continue.

Additional check: tests with literal > in their names.

After offload collect passes, run npx vitest list --json and check for test names where a >-separated segment itself contains > (e.g. describe('parse') > it('attribute value with >')). Offload uses > as the separator between the file path and the describe/test chain in test IDs, so a literal > in a test name creates ambiguity that causes "Not Run" results.

If any are found, present them to the user and recommend renaming the affected test()/it()/describe() calls to avoid > characters. For example, rename 'attribute value with >' to 'attribute value with greater-than'. This is not a blocking error — offload collect will still pass — but affected tests will be reported as "Not Run" after execution because Offload cannot match their results back to discovered IDs.

Step 6: Create Local Invocation Script

Create scripts/offload-tests.sh:

#!/usr/bin/env bash
#
# Run the project's test suite via Offload (parallel on Modal).
# Requires: Offload (cargo install offload), Modal CLI + credentials
#
set -euo pipefail

if ! command -v offload &> /dev/null; then
    echo "Error: 'offload' not installed. Install with: cargo install offload"
    exit 1
fi

cd "$(git rev-parse --show-toplevel)"
exec offload run --copy-dir ".:/app" "$@"

Make it executable with chmod +x scripts/offload-tests.sh.

The --copy-dir flag tells Offload to bake the local directory into the sandbox image at the given path during the prepare step. The target path must match sandbox_repo_root in offload.toml (e.g. ".:/app" when sandbox_repo_root = "/app"). This is specified at invocation time, not in offload.toml, because it depends on where you're running from.

Use this script (or the equivalent invocation) for all subsequent steps that run Offload.

If the project uses a Makefile, justfile, or Taskfile instead of scripts/, add the invocation there instead to be consistent with existing practice.

Step 7: Update .gitignore

Append Offload artifacts to .gitignore:

# Offload
test-results/

NOTE: Do not confuse .gitignore (which controls what git tracks) with .dockerignore (which controls what gets copied into the sandbox image). The .dockerignore is only created if needed during troubleshooting — see the Troubleshooting section.

Step 8: Run Offload Locally and Verify

Install offload if not already present:

cargo install offload

Run the tests using the invocation script from Step 6:

First, verify test discovery with offload collect:

offload collect

This runs test discovery locally without creating sandboxes or executing tests. Fix any errors until offload collect succeeds and lists the expected tests. Common issues:

1. "No tests discovered": Check that paths in offload.toml points to the correct test directories and the framework command is correct.
2. "Duplicate test names found" (vitest): Duplicate space-separated test names exist. Return to Step 5 and resolve them.
3. Discovery command failed: The framework tool (pytest, cargo nextest, npx vitest) is not installed or not on PATH.

Do not proceed to execution until offload collect lists the expected tests.

Then, establish a baseline by running the test suite directly (without Offload):

# For vitest:
pnpm exec vitest run --project <project>
# For pytest:
<command> --tb=short
# For cargo:
cargo nextest run

Record the number of passed, failed, and skipped tests. Some projects have pre-existing test failures (snapshot mismatches, missing fixtures, flaky tests). These are not your problem to fix — the goal is for Offload's results to match this baseline, not to achieve zero failures.

Then, run the full test suite via Offload:

./scripts/offload-tests.sh

Compare Offload's results against the baseline. Offload is working correctly when:

• The number of passed tests matches the baseline (within 1-2 for flaky tests)
• The number of failed tests matches the baseline
• There are no "Not Run" tests (0 missing)

If Offload's results differ from the baseline, iterate until they match. Common issues and fixes:

1. "Exec format error": .venv or local binaries leaked into the sandbox. See the Troubleshooting section on creating a .dockerignore.
2. "No such file or directory": The sandbox is missing a dependency. Check the Dockerfile has the right runtime and package manager.
3. "Token validation failed": Modal credentials are expired. Run modal token new to refresh.
4. Tests discovered but "Not Run": The test command is failing silently inside the sandbox. Check batch logs in test-results/logs/ for errors. Debug by checking if uv/python/cargo is available in the Dockerfile.
5. More failures than baseline: Offload may be running tests in a different order or with different environment. Check batch logs for the extra failures and compare with the baseline output.
6. Fewer tests discovered than expected: Check that offload collect finds all the tests. If using vitest with --project, ensure the filter matches.

Keep running and fixing until Offload's pass/fail counts match the baseline. Do not proceed to optimization until they match.