Local Platform E2E

Local end-to-end: @benchsdk/runner ↔ benchmarks-platform

Goal: exercise upsert benchmark → create run → planWorkers → claim → heartbeat → task_results → artifact upload → complete → dashboard, with zero external credentials.

1. Infra (docker)

docker run -d --name pg -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=bench postgres:16
docker run -d --name minio -p 9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  quay.io/minio/minio server /data
docker run -d --name ch -p 8123:8123 -e CLICKHOUSE_PASSWORD=chpass clickhouse/clickhouse-server:25.6
docker run --rm --network host --entrypoint sh quay.io/minio/mc -c \
  "mc alias set local http://127.0.0.1:9000 minioadmin minioadmin && mc mb -p local/bench"

• Use ClickHouse >= 25.x: 24.8 fails ch:migrate with TTL expression result column should have DateTime or Date type, but has DateTime64(3,'UTC').
• The importer passes clickhouse_settings: { date_time_input_format: 'best_effort' } per insert, so a default-configured server works. If you hit Cannot parse input: expected '"' before: 'Z"...' (older code), work around it server-side, and remember to REMOVE the override before verifying an importer fix — otherwise the server setting masks it:

  docker exec ch bash -c 'mkdir -p /etc/clickhouse-server/users.d && printf "<clickhouse><profiles><default><date_time_input_format>best_effort</date_time_input_format></default></profiles></clickhouse>" > /etc/clickhouse-server/users.d/besteffort.xml'
  docker restart ch
  # verify which mode is actually active:
  curl -s "http://127.0.0.1:8123/?user=default&password=chpass" \
    --data-binary "SELECT value FROM system.settings WHERE name='date_time_input_format'"

2. benchmarks-platform .env.local

Point TIGRIS_* at MinIO — the events and artifacts routes require S3 or they return 502 on every batch. region: "auto" + presigned PUT works with MinIO.

DATABASE_URL=postgresql://postgres:postgres@127.0.0.1:5432/bench
DATABASE_URL_UNPOOLED=postgresql://postgres:postgres@127.0.0.1:5432/bench
CLICKHOUSE_URL=http://127.0.0.1:8123
CLICKHOUSE_DATABASE=default
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=chpass
ADMIN_API_KEY=local-admin-key
BETTER_AUTH_SECRET=<openssl rand -hex 32>
BETTER_AUTH_URL=http://localhost:3000
TIGRIS_ACCESS_KEY_ID=minioadmin
TIGRIS_SECRET_ACCESS_KEY=minioadmin
TIGRIS_STORAGE_ENDPOINT=http://127.0.0.1:9000
TIGRIS_BUCKET=bench

Then:

npm run db:migrate
CLICKHOUSE_URL=http://127.0.0.1:8123 CLICKHOUSE_USER=default CLICKHOUSE_PASSWORD=chpass npm run ch:migrate
npm run dev

ch:migrate does not read .env.local; without exported vars it silently prints "CLICKHOUSE_URL not set; skipping".

3. Mandatory seed (otherwise every run creation 500s on an FK)

app/api/v1/benchmarks/[slug]/runs/route.ts hardcodes a default org and user id for every created run. Insert those exact rows:

INSERT INTO "user" (id,name,email,email_verified)
VALUES ('mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG','David Tice','david@example.com',true);
INSERT INTO organization (id,name,slug,created_at,owner_id)
VALUES ('zMrSfAyEyVJ2eKIxgoBZtrMvEd6a78ad','ComputeSDK','computesdk',now(),'mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG');
INSERT INTO member (id,organization_id,user_id,role,created_at)
VALUES ('mem1','zMrSfAyEyVJ2eKIxgoBZtrMvEd6a78ad','mxYI5c90QNkRPhvuc2HvNHy5mM7MG7jG','owner',now());

Re-read those constants before seeding — they may change.

As of the org-scoped-auth change (lib/api/api-auth.ts requireApiAuth), this seed is only needed for the admin-key path: an org-scoped key supplies the run's organizationId itself and userId from the key's created_by. Admin-key runs with no organizationId in the body still fall back to those two hardcoded ids.

3b. Minting org-scoped bp_ API keys locally

The org API-key HTTP route is session-scoped and answers {"error":"Unauthenticated"} to the admin key, so keys can only be created from the dashboard UI or directly. For scripted multi-tenant tests, mint them with the platform's own generator so the sha256 hash/prefix/lastFour match what verifyApiKey() expects:

// scripts/e2e-seed-orgs.ts (throwaway), run with:
//   npx tsx --env-file=.env.local scripts/e2e-seed-orgs.ts
import { generateApiKey } from "@/lib/api-keys";
import { apiKey, member, organization, user } from "@/db/auth-schema";
const g = generateApiKey();  // g.plaintext is the bp_<prefix>_<secret> to send
await db.insert(apiKey).values({
  id, organizationId, name, prefix: g.prefix, hashedKey: g.hashedKey,
  lastFour: g.lastFour, createdBy: someUserId, createdAt: new Date(),
  revokedAt: null, expiresAt: null,   // set these to test revoked/expired → 401
});

Each org needs user + organization (owner_id) + member rows first. To view an org's runs in the dashboard, add a member row for your dashboard user in that org — dashboard auth is session-based and completely separate from API keys.

4. Dashboard access

Sign up via POST /api/auth/sign-up/email (email+password is enabled), then add a member row for that user in the seeded org, and sign in at http://localhost:3000/signin. Run pages live at /{orgSlug}/benchmarks/{benchmarkSlug}/runs/{runId} (+ /workers).

5. Getting results into ClickHouse locally

@vercel/queue.send() fails locally (swallowed as a warning), so nothing imports automatically. Trigger it by hand — the cron route accepts the admin key when CRON_SECRET is unset:

curl -s "http://localhost:3000/api/cron/import-clickhouse?limit=50" -H "Authorization: Bearer local-admin-key"

Check imported/failed/failureSamples in the response.

6. Running a benchmark with no provider credentials

Build first (packages/*/dist is not committed): pnpm install && pnpm -r --filter "./packages/**" build.

Write a throwaway bench inside the repo (untracked, e.g. e2e-local/local.bench.ts) so pnpm workspace resolution finds @benchsdk/runner, with a fake participant:

import { defineBenchmarkConfig, defineTask } from '@benchsdk/runner';
export const config = defineBenchmarkConfig({
  benchmarkSlug: 'e2e-local', benchmarkName: 'E2E', iterations: 4, concurrency: 1,
  participants: [{ name: 'local', requiredEnvVars: [] }],
});
export const task = defineTask(async (ctx) => {
  await ctx.step('create', () => new Promise((r) => setTimeout(r, 50)));
  ctx.measure({ ok: true });
});

Run it via the bench run CLI (under tsx so the .bench.ts module loads without a build):

BENCHMARKS_PLATFORM_URL=http://localhost:3000 BENCHMARKS_PLATFORM_API_KEY=<org bp_ key> \
  npx tsx packages/benchsdk-runner/dist/bin.js run e2e-local/local.bench.ts --iterations 4 --concurrency 2

BENCHMARKS_PLATFORM_URL is the root URL (the runner appends /api/v1).

The runner authenticates with BENCHMARKS_PLATFORM_API_KEY (an org-scoped bp_ key — mint one locally per the section below) and pulls the owning org slug from the server, so the "View at:" URL it prints always points at the run's real org (no BENCHMARKS_PLATFORM_ORG_SLUG needed).

6b. Probing tenant isolation

With benchmarks-platform#47, all app/api/v1/benchmarks/** routes take either the global ADMIN_API_KEY or an org bp_ key. Expected shapes when a key from org B addresses org A's run: 403 {"error":"API key is not authorized for this organization"} (getScopedRun / requireRunAccess), 404 for an unknown run/benchmark, 401 Invalid API key for revoked/expired/malformed tokens, 401 API key required with no header. Listings (GET /benchmarks/:slug/runs and /results) narrow by organizationScope(auth).

When probing the heartbeat in-process cache, the body must carry currentStep, all four progressDone/InFlight/Errors/Total fields and concurrency (top-level, not nested) or the cache is never populated and your "cache poisoning" probe proves nothing. A cache-served beat is recognisable because the response's worker object is minimal (no status/benchmarkId columns). Route timing metadata (cacheCoalesced) is only logged for requests slower than 1000 ms, so don't rely on the dev-server log for this.