Ab Test Zephyr

A/B test Zephyr changes

Signals

The coordinator writes one zephyr.stage row per completed stage and execution_id. Use these fields:

cpu_time_total sums process user and system CPU-seconds across completed shards, so worker count and queue delay do not directly change it. Use it as the primary efficiency signal and normalize per item or byte when accepted workload sizes differ. elapsed measures the stage barrier and includes startup, I/O, concurrency, queueing, and stragglers; repeat an elapsed-only result under comparable scheduling conditions.

Keep CPU and elapsed time as separate outcomes:

• CPU flat or lower and elapsed lower: latency or topology win without added compute cost.
• CPU higher and elapsed lower: faster and more expensive.
• CPU lower and elapsed higher: cheaper and slower.
• Wall-only change from one comparison: inconclusive until repeated.
• Topology or batching change: report the latency/compute tradeoff; do not describe wall-time gains as equivalent per-core efficiency gains.

Calibrate thresholds from same-code repeats for the selected sample and pool shape. A new OOM, application failure, or memory peak above the worker limit is a regression regardless of CPU.

Choose the comparison

Existing runs

Start at Collect execution IDs when control and treatment jobs already exist. Confirm that the control and every treatment used the same immutable sample, stage range, sources, worker resources, concurrency, parallelism, cluster, region, and priority.

A scheduled baseline is usable only when its report contains the same workload fingerprint and its Finelog execution IDs remain queryable. Otherwise, launch a matching control. Do not compare a standalone benchmark treatment with a differently shaped ferry baseline.

New runs

Default to one control at the branch/PR merge base and one treatment at the branch/PR head. Add treatments only when the requester explicitly names each additional commit or configuration. Record a stable name plus the exact SHA and configuration difference for every extra arm; do not infer or invent arms.

Run on GCP in europe-west4 with gs://marin-eu-west4/datakit/sample_100b_8ae7a94f unless the requester selects another sample or backend. The us-central1 GCS sample is available for us-central1 runs. CoreWeave remains available for S3-local runs; select it explicitly with the matching S3 sample and target cluster.

For a PR, read the diff and select the smallest stage range that exercises the changed behavior:

Confirm the sample size, pool shape, stage range, cluster, and expected cost before launching an expensive or production-scale comparison. Run local Zephyr and Datakit tests before paying for remote workers.

Prepare worktrees

For a PR, use the merge base as the control and the PR head as the first treatment:

git fetch origin main
BASELINE_SHA=$(git merge-base origin/main HEAD)
TREATMENT_SHA=$(git rev-parse HEAD)
WORKTREE_ROOT=$(mktemp -d /tmp/zephyr-ab.XXXXXX)
git worktree add --detach "$WORKTREE_ROOT/control" "$BASELINE_SHA"
git worktree add --detach "$WORKTREE_ROOT/treatment" "$TREATMENT_SHA"

Record both SHAs. Preserve configuration-only arms in separate worktrees or commits. Add arms only when explicitly requested.

Launch the download-free benchmark

experiments.datakit.zephyr_benchmark accepts an existing normalized sample and routes outputs to a seven-day temporary prefix. Use an immutable, region-local sample. Its default input is the GCS 100B sample in europe-west4. All arguments except --run-tag must match across the control and treatments.

Set exactly one data-locality argument before launching:

# Default: GCS input and GCP compute in europe-west4.
SAMPLE_PREFIX=gs://marin-eu-west4/datakit/sample_100b_8ae7a94f
DATA_LOCALITY_ARGS=(--region europe-west4)

# GCP opt-in: use the existing us-central1 sample with us-central1 compute.
# SAMPLE_PREFIX=gs://marin-us-central1/datakit/sample_100b_8ae7a94f
# DATA_LOCALITY_ARGS=(--region us-central1)

# CoreWeave opt-in: S3 input and CoreWeave compute in cw-us-east-02a.
# SAMPLE_PREFIX=s3://marin-us-east-02a/marin/datakit/sample_100b_8ae7a94f
# DATA_LOCALITY_ARGS=(--target-cluster cw-us-east-02a)

Set the cluster or region from the actual sample prefix. If the mapping is unknown, stop before launching. The benchmark passes source_prefix to marin_temp_bucket, which keeps temporary outputs with the sample. Do not override the output location or launch compute in a different region.

Launch each arm from its worktree:

cd <CONTROL_OR_TREATMENT_WORKTREE>
uv run iris --config=lib/iris/config/marin.yaml job run --no-wait \
  --job-name zephyr-ab-<RUN_TAG>-<ARM> \
  "${DATA_LOCALITY_ARGS[@]}" --memory=2G --disk=5G --cpu=1 --extra=cpu \
  --priority batch \
  -- python -m experiments.datakit.zephyr_benchmark \
    --sample-prefix "$SAMPLE_PREFIX" \
    --sources <COMMA_SEPARATED_SOURCES_OR_ALL> \
    --run-tag <FRESH_RUN_TAG>-<ARM> \
    --pool-workers <WORKERS> \
    --pool-cpu <CPU_PER_WORKER> \
    --pool-ram <RAM_PER_WORKER> \
    --pool-disk <DISK_PER_WORKER> \
    --first-stage <exact|tokenize|minhash|fuzzy> \
    --last-stage <exact|tokenize|minhash|fuzzy> \
    --max-concurrent <PIPELINES> \
    --dedup-max-parallelism <SHARDS>

Record this workload fingerprint for every arm:

• commit SHA and Iris job ID
• sample prefix and source selection
• first and last stage
• pool workers, CPU, RAM, and disk
• maximum concurrent pipelines and dedup parallelism
• Iris controller, data-local target cluster or region, priority, and preemptibility
• run tag

Use fresh run tags so no arm cache-hits. One matching control can be reused for explicitly requested treatments launched in the same scheduling window. If the decision depends on elapsed time, interleave additional control trials among the treatments to measure scheduling noise.

If the request includes continuous monitoring, use babysit-zephyr. A failed or preempted arm measures infrastructure reliability and carries no performance result. Use debug only for a stated repeated fault.