Device Mode E2E

Device-mode e2e harness

Objective: remove the manual effort of hand-writing an HTML page, opening it, and reading the Network tab to check whether a device-mode integration change actually forwards events. Given a write key alone, this harness builds and serves the SDK, fires targeted events at the real connection, and reports a generic PASS/FAIL. It is destination-agnostic: no integration-specific verification is baked in.

What it verifies (and what it does not)

Generic, automated (any device-mode integration):

• the SDK fetched the real sourceConfig from the write key and got a 2xx,

• it loaded the device-mode integration bundle (/js-integrations/),

• ready() fired and every event was forwarded without throwing,

• DATA was actually sent to a third-party (non-localhost, non-RudderStack) host — a fetch/XHR/ beacon/pixel, not merely a native-SDK <script> load. "native SDK asset loaded" is reported separately as info; only "data sent" counts as delivery.

• the destination config arrived in the SDK with the shape the control-plane contract says — for each key rudder-integrations-config lists for a web device-mode connection and that was delivered, the value has the expected type. A key stored per source type ({ web: … }) must arrive resolved to its inner value, not as an object; this catches delivery bugs the SDK would otherwise swallow. Reported as warnings by default; configContract: "strict" fails the run on a type mismatch. Keys that are listed but absent are reported as missing and are informational only — most destination settings are optional, so an unset one legitimately never reaches the SDK. Keys delivered but not listed appear as unexpected, also informational. Schema conformance is not checked here — that is rudder-integrations-config's own CI job.

Captured console/page errors (console.error, onerror, unhandledrejection) are shown as warnings — they do not fail the run (third-party SDKs log noise).

It also SHOWS you the actual outgoing requests — method, URL, query params, and body — so you can confirm your fix put the right value in the payload (say, the SDK's messageId arriving as the destination's own event id), not just that a request fired. You can optionally add generic expectations to turn a specific value check into an automated PASS/FAIL (see step 5 and step 7).

Manual (out of scope — inherently per-destination): confirming the events arrived inside the destination's dashboard. The harness ends by reminding you to check it. Do not try to automate this in the generic skill.

FIRST: ask the user two things (do not skip, do not choose for them)

Before building or running anything, ask the user and wait for their answers:

1. Which CDN to load the SDK from?

• local (default) — the localhost dev servers, i.e. your uncommitted code. Requires the build + serve steps.
• staging (https://cdn.staging.rudderlabs.com) / production (https://cdn.rudderlabs.com) / dev (https://cdn.dev.rudderlabs.com) — the released SDK for that write key. No build or serve needed — skip steps 2 & 3 (the step-1 preflight still applies). Use this to reproduce a customer issue or verify a config-only change against live code.
• a specific/custom CDN — any https://… base; ask if they want one.

Present those defaults and let them pick (or name a custom base). Set cdn (and optional variant: modern/legacy) in the config accordingly; local keeps the explicit localhost URLs.

2. Which verification mode?

• Interactive — you build/serve (or use the CDN), open the harness page in their browser, and they click ▶ Run tests and inspect the report. Best for onboarding / large changes.
• Report — you run it automatically and hand back the report (test cases used, actual outgoing requests, and an "expected on the <destination> dashboard" summary). Best for minimal / targeted changes (e.g. eventId-from-messageId).

You may recommend based on the change ("minimal → Report", "config-only → production CDN"), but the user decides. Only skip a question if they already answered it (or said "just pick"). Do not silently default to Report, do not run headless without being asked, and do not assume local if they might want to test the released SDK — surface the choice.

Inputs

• Integration — the folder name under packages/analytics-js-integrations/src/integrations/. Used to locate the source for test-case derivation and to match the loaded bundle name. Never used for verification logic.
• writeKey (required) — a workspace source that has this integration connected as a device-mode destination. The SDK resolves the real connection from this alone.
• SDK version — v3 (default) or v1.1.

Prerequisites

• The repo's Node version — pinned in .nvmrc at the repo root. The headless runner needs the global WebSocket (Node ≥ 22), and your shell may default to an older Node; switch first and verify:

  nvm use && node -v      # picks up .nvmrc; or point the scripts at an explicit Node >= 22 binary

• Chrome/Chromium installed. The runner probes standard locations; if it can't find yours, set CHROME_PATH. On macOS the default is:

  export CHROME_PATH="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"

If Node ≥ 22 or Chrome is unavailable, use the zero-dep fallback: serve the directory that holds only the generated page (<temp>/page, see step 6) with any static server, open it in your own browser, click ▶ Run tests, and read the on-page PASS/FAIL table — no runner needed. (run-cdp.mjs is only for the automated report flow.)

Never point a directory server at the dir holding the run config or preflight.json. A native destination SDK loaded by the page runs in the page's origin, so it could fetch those sibling files and read the write key / destination config. run-cdp.mjs serves the single HTML file for exactly this reason; a plain static server does not, so give it a directory with nothing else in it.

Steps

1. Preflight — confirm the write key actually has the integration connected

Do this first — it catches "zero destinations" in seconds and is cache-busted (so it reflects a just-connected destination, not a stale control-plane copy):

node .claude/skills/device-mode-e2e/harness/preflight.mjs \
  --writeKey <WRITE_KEY> --integration <Integration> --json <temp>/preflight.json

Exit 0 = connected & enabled. Exit 1 = not connected / disabled / zero destinations — fix the connection first. (Add --configUrl for a non-default control plane.)

--json writes the resolved destinations — id, enabled, connectionMode and their live config. It is required only for a config override (step 5), but always worth passing: the printed id is what an override is keyed by.

Use this layout for the temp files (it keeps secrets off the page's own origin — see Security):

<temp>/preflight.json     # destination config — NOT served
<temp>/run.json           # run config, embeds the write key — NOT served
<temp>/page/harness.html  # the only file any server is pointed at

2. Build + serve the SDK, integrations, AND plugins — only for cdn: local

Skip this entire step (and step 3) when the user chose a CDN (staging/production/dev/custom) — the SDK, integrations, and plugins all load from that CDN, so there's nothing to build or serve. Go straight to derive test cases (step 4) → config with cdn (step 5) → run.

A v3 local run needs three servers. Run each in its own terminal (they keep watching/serving). Use the core/v1.1 :no-open scripts so their dev servers serve without opening a demo tab on top of the harness window (plain start/start:modern still open one):

# 1) core SDK   → http://localhost:3001  (rsa.min.js under /cdn/<variant>/iife/)
cd packages/analytics-js && npm run start:modern:no-open

# 2) integrations → http://localhost:3005  (bundles under /cdn/<variant>/js-integrations/)
cd packages/analytics-js-integrations && npm run start:modern

# 3) plugins    → http://localhost:3002  (rsa-plugins.js under /cdn/<variant>/plugins/)
cd packages/analytics-js-plugins && npm run start:modern

Why the plugins server is required (this is the #1 gotcha). With the CDN snippet, when pluginsSDKBaseURL is omitted the SDK does not fall back to the public CDN — it derives the plugins URL from where the core script was loaded (localhost:3001), which does not serve plugins. Every plugin then silently fails to load — including the device-mode-destinations plugin, so no integration bundle is ever requested. You must serve the plugins package and set pluginsSDKBaseURL. (Same mechanism applies to destSDKBaseURL for integrations.)

Faster iteration: the integrations start rebuilds every bundle (minutes). To rebuild just one, use the targeted CLI build, then serve dist:

cd packages/analytics-js-integrations
BROWSERSLIST_ENV=modern npm run build:integration:cli --intg=<Integration>
npx serve ./dist -p 3005 --cors

Base URLs for a v3 modern build (use legacy instead of modern for a legacy build):

• sdkUrl = http://localhost:3001/cdn/modern/iife/rsa.min.js
• destSDKBaseURL = http://localhost:3005/cdn/modern/js-integrations
• pluginsSDKBaseURL = http://localhost:3002/cdn/modern/plugins

For v1.1, serve the legacy core (cd packages/analytics-v1.1 && npm run start:no-open; bundle rudder-analytics.min.js) and keep the integrations server; v1.1 does not use the remote plugins. Confirm ports/filenames from each dev-server log — they can differ from your setup.

3. Wait for the servers to be ready (local only)

The dev servers are not ready immediately (the integrations start builds everything first). Poll until all three bundles return 200 before running:

for url in \
  http://localhost:3001/cdn/modern/iife/rsa.min.js \
  http://localhost:3005/cdn/modern/js-integrations/<Integration>.min.js \
  http://localhost:3002/cdn/modern/plugins/rsa-plugins.js ; do
  until curl -sf -o /dev/null "$url"; do echo "waiting: $url"; sleep 3; done
done
echo "all servers ready"

4. Derive targeted test cases from the implementation (the LLM step)

Read the integration source and the change under test, then synthesize a small, targeted set of events — do not blast every event type:

# the implementation
packages/analytics-js-integrations/src/integrations/<Integration>/browser.js   # + utils.js, constants.js, nativeSdkLoader.js
# what changed
git -C <repo> diff -- packages/analytics-js-integrations/src/integrations/<Integration>/

Decide which SDK methods the integration implements (identify, track, page, group, alias) and which the change touches, and craft events exercising exactly those paths and payload shapes. Examples: trait handling changed → an identify with those traits; a new track property mapping → a track carrying that property; page-name handling → a page call.

5. Write the run config

Create a JSON config (full schema in harness/README.md). Write it to a temp dir outside the repo — it contains the write key in plaintext (see Security below):

cdn: local (build + serve) — give the three localhost URLs:

{
  "writeKey": "<WRITE_KEY>",
  "integration": "<Integration>",
  "sdkVersion": "v3",
  "cdn": "local",
  "sdkUrl": "http://localhost:3001/cdn/modern/iife/rsa.min.js",
  "destSDKBaseURL": "http://localhost:3005/cdn/modern/js-integrations",
  "pluginsSDKBaseURL": "http://localhost:3002/cdn/modern/plugins",
  "settleMs": 6000,
  "events": [
    /* the targeted events derived in step 4 */
  ],
  "expectations": [
    /* optional — see below */
  ]
}

cdn: staging | production | dev | <https base> — omit the three URLs; generate.mjs derives them as <base>/v3/<variant>/{rsa.min.js, js-integrations, plugins} (variant defaults to modern):

{
  "writeKey": "<WRITE_KEY>",
  "integration": "<Integration>",
  "cdn": "production",
  "settleMs": 6000,
  "events": [
    /* … */
  ]
}

Explicit sdkUrl/destSDKBaseURL/pluginsSDKBaseURL always override the derived ones. CDN auto-URLs are v3-only; for v1.1 on a CDN, set them explicitly. Leave dataPlaneUrl unset unless you also want cloud-mode. Set configUrl only for a non-default control plane (EU / self-hosted / staging CP).

Optional expectations turn a specific value check on the outgoing request into an automated PASS/FAIL (generic — no per-destination logic). Use them when the change is about what the request carries:

// e.g. a fix that must forward a value into a specific request field
{ "description": "<what this proves>", "requestIncludes": "<expected value>" }
{ "description": "<what this proves>", "requestBodyPath": "<field>", "equals": "<value>" }

Both are evaluated only against DATA requests made after the events fired (a pre-event startup request can't satisfy them). requestIncludes matches a substring in any such request's URL or body; requestBodyPath+equals checks a JSON body field with strict === (so 1 ≠ "1" — match the type). For a user-controlled field, set a recognizable sentinel value in the event and assert it. For an SDK-generated id (messageId), you can't predict it — rely on the printed request body (step 7) instead of a static expectation.

Optional configOverride varies the destination config for one run — without editing the dashboard. Use it to exercise a config-dependent code path (a version switch, a new setting, a mapping toggle) across several runs against one connection:

{
  // …the fields above, plus:
  "configOverride": { "<configKey>": "<value>" },
  "destinationId": "<id>", // from step 1; or…
  "preflightJson": "<temp>/preflight.json", // …let the id be resolved from the snapshot
}

• Write it in the shape the SDK receives, i.e. values already resolved for the web source type: "<configKey>": "<value>", not { "<configKey>": { "web": "<value>" } }. The override merges into the config the integration reads, not into the control plane's stored form. Keys and values are whatever the destination under test defines — run dest-config.mjs (below) to list them.
• It becomes the SDK's v3 sourceConfigurationOverride load option, so the real sourceConfig call still happens and is still verified — only the matched destination's config is adjusted afterwards.
• The SDK shallow-merges it ({...real, ...override}), so a nested object is replaced wholesale — pass the whole object.
• v3 only. sourceConfigurationOverride does not exist in v1.1; the combination is rejected.
• It needs the destination id: give destinationId, or preflightJson to resolve it by name.
• generate.mjs warns when an overridden key is not one the control plane sends for a web device-mode connection (db-config.json's destConfig.web + destConfig.defaultConfig) — such an override does not mirror production.
• The delivered-config check reports what the control plane delivered, before your override is applied — so a mismatch there is about the real connection, not about what you overrode.

To see a destination's expected web shape, or to check a delivered config offline:

# expected shape (which keys, which delivered type)
node .claude/skills/device-mode-e2e/harness/dest-config.mjs --integration <Integration>
# compare a delivered config (preflight --json, or result.deliveredConfig) against it
node .claude/skills/device-mode-e2e/harness/dest-config.mjs \
  --integration <Integration> --config <delivered-config.json> [--strict]   # --remote reads GitHub

The contract is read from a local checkout (integrationsConfigPath, env INTEGRATIONS_CONFIG_PATH, else ~/workspace/rudder-integrations-config), falling back to GitHub (integrationsConfigRef, default develop; integrationsConfigRemote: true forces it). With no contract available the run simply skips the check and says so.