Hardening Flaky E2E Tests
Comfy-Org/ComfyUI_frontend
Diagnoses and fixes flaky Playwright e2e tests by replacing race-prone patterns with retry-safe alternatives.
Write and run Playwright E2E tests for Supabase Studio (e2e/studio).
$ npx skills add supabase/supabase --skill studio-e2e-tests -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install supabase/supabase studio-e2e-tests --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/supabase/supabase.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/studio-e2e-tests .claude/skills/studio-e2e-tests && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "studio-e2e-tests" agent skill from https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-tests into .claude/skills/studio-e2e-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "studio-e2e-tests", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-testsType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add supabase/supabase --skill studio-e2e-tests -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install supabase/supabase studio-e2e-tests --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/supabase/supabase.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/studio-e2e-tests .agents/skills/studio-e2e-tests && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "studio-e2e-tests" agent skill from https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-tests into .agents/skills/studio-e2e-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "studio-e2e-tests", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add supabase/supabase --skill studio-e2e-tests -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install supabase/supabase studio-e2e-tests --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/supabase/supabase.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/studio-e2e-tests .cursor/skills/studio-e2e-tests && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "studio-e2e-tests" agent skill from https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-tests into .cursor/skills/studio-e2e-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "studio-e2e-tests", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/supabase/supabase.git --path .agents/skills/studio-e2e-tests--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add supabase/supabase --skill studio-e2e-tests -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install supabase/supabase studio-e2e-tests --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/supabase/supabase.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/studio-e2e-tests .gemini/skills/studio-e2e-tests && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "studio-e2e-tests" agent skill from https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-tests into .gemini/skills/studio-e2e-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "studio-e2e-tests", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install supabase/supabase studio-e2e-testsInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add supabase/supabase --skill studio-e2e-tests -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/supabase/supabase.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/studio-e2e-tests .github/skills/studio-e2e-tests && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "studio-e2e-tests" agent skill from https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-tests into .github/skills/studio-e2e-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "studio-e2e-tests", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add supabase/supabase --skill studio-e2e-tests -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install supabase/supabase studio-e2e-tests --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/supabase/supabase.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/studio-e2e-tests .opencode/skills/studio-e2e-tests && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "studio-e2e-tests" agent skill from https://github.com/supabase/supabase/tree/master/.agents/skills/studio-e2e-tests into .opencode/skills/studio-e2e-tests/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "studio-e2e-tests", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
studio-e2e-testsWrite and run Playwright E2E tests for Supabase Studio (e2e/studio).
Studio E2E Tests is an agent skill from supabase/supabase, published by the product's own GitHub organization. Write and run Playwright E2E tests for Supabase Studio (e2e/studio). Use when asked to run e2e tests, write new E2E tests, or debug flaky or failing Playwright tests. Covers running commands, avoiding race conditions, waiting strategies, selectors, helper functions, and CI vs local differences.
Its SKILL.md is about 2.8k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in Testing & QA, covering End-to-end testing, Async programming and Browser testing. It works with Supabase and Playwright. The repository describes itself as: The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications. The licence is Apache-2.0.
4 steps, taken from the first numbered list in SKILL.md.
Read from SKILL.md and the folder at commit 26c838a. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
pnpmFrom the folder's file list and the shell code blocks in SKILL.md.
No URLs in SKILL.md. Its commands use pnpm, which can reach the network depending on how they are called.
From URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Studio E2E Tests loads about 2.8k tokens when it runs. Until then it costs about 78 tokens; SKILL.md has 669 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from supabase/supabase at commit 26c838a, republished under its Apache-2.0 licence (© supabase). 669 words, ~2,762 tokens.
.claude/skills/studio-e2e-tests/SKILL.md (or your agent's skills folder).Run Playwright end-to-end tests for the Studio application.
Tests must be run from the e2e/studio directory:
cd e2e/studio && pnpm run e2ecd e2e/studio && pnpm run e2e -- features/cron-jobs.spec.tscd e2e/studio && pnpm run e2e -- --grep "test name pattern"cd e2e/studio && pnpm run e2e -- --uiIS_PLATFORM=false) runs tests in parallel (3 workers)e2e/studio/features/*.spec.tsimport { test } from '../utils/test.js'page, ref, and other helpersWait for elements with generous timeouts:
await expect(locator).toBeVisible({ timeout: 30000 })Use serial mode for tests sharing database state:
test.describe.configure({ mode: 'serial' })getByRole with accessible name - Most robust, tests accessibility
page.getByRole('button', { name: 'Save' })
page.getByRole('button', { name: 'Configure API privileges' })getByTestId - Stable, explicit test hooks
page.getByTestId('table-editor-side-panel')getByText with exact match - Good for unique text
page.getByText('Data API access', { exact: true })locator with CSS - Use sparingly, more fragile
page.locator('[data-state="open"]')XPath selectors - Fragile to DOM changes
// BAD
locator('xpath=ancestor::div[contains(@class, "space-y")]')Parent traversal with locator('..') - Breaks when structure changes
// BAD
element.locator('..').getByRole('button')Broad filter({ hasText }) on generic elements - May match multiple elements
// BAD - popover may have more than one combobox
// Could consider scoping down the container or filtering the combobox more specifically
popover.getByRole('combobox')When a component lacks a good accessible name, add one in the source code:
// In the React component
<Button aria-label="Configure API privileges">
<Settings />
</Button>Then use it in tests:
page.getByRole('button', { name: 'Configure API privileges' })Scope selectors to specific containers to avoid matching wrong elements:
// Good - scoped to side panel
const sidePanel = page.getByTestId('table-editor-side-panel')
const toggle = sidePanel.getByRole('switch')
// Good - find unique element, then scope from there
const popover = page.locator('[data-radix-popper-content-wrapper]')
const roleSection = popover.getByText('Anonymous (anon)', { exact: true })Set up API waiters BEFORE triggering actions. This is the most common source of flaky tests.
// ❌ Race condition — response may complete before waiter is set up
await page.getByRole('button', { name: 'Save' }).click()
await waitForApiResponse(page, 'pg-meta', ref, 'query?key=table-create')
// ✅ Waiter is ready before the action
const apiPromise = waitForApiResponse(page, 'pg-meta', ref, 'query?key=table-create')
await page.getByRole('button', { name: 'Save' }).click()
await apiPromiseSame rule applies before navigation:
const loadPromise = waitForTableToLoad(page, ref)
await page.goto(toUrl(`/project/${ref}/editor?schema=public`))
await loadPromiseWhen an action triggers multiple API calls, wait for all of them:
const createTablePromise = waitForApiResponseWithTimeout(page, (r) =>
r.url().includes('query?key=table-create')
)
const tablesPromise = waitForApiResponseWithTimeout(page, (r) =>
r.url().includes('tables?include_columns=true')
)
await page.getByRole('button', { name: 'Save' }).click()
await Promise.all([createTablePromise, tablesPromise])Playwright auto-waits for elements to be actionable — prefer this over manual timeouts.
Use expect.poll for dynamic state changes:
await expect.poll(async () => await page.getByLabel(`View ${tableName}`).count()).toBe(0)Use waitForSelector with state for element lifecycle:
await page.waitForSelector('[data-testid="side-panel"]', { state: 'detached' })Avoid networkidle — use specific API waits instead:
// ❌ Unreliable and slow
await page.waitForLoadState('networkidle')
// ✅ Specific API response
await waitForApiResponse(page, 'pg-meta', ref, 'tables')The only acceptable use of waitForTimeout is a client-side debounce:
await page.getByRole('textbox').fill('search term')
await page.waitForTimeout(300) // allow debouncewaitForTimeoutNever use waitForTimeout to wait for UI or network — always wait for something specific (the debounce case above is the sole exception):
// BAD
await page.waitForTimeout(1000)
// GOOD - wait for UI element
await expect(page.getByText('Success')).toBeVisible()
// GOOD - wait for API response
const apiPromise = waitForApiResponse(page, 'pg-meta', ref, 'query?key=table-create')
await saveButton.click()
await apiPromise
// GOOD - wait for toast indicating operation complete
await expect(page.getByText('Table created successfully')).toBeVisible({ timeout: 15000 })force: true on clicksInstead of forcing clicks on hidden elements, make them visible first:
// BAD
await menuButton.click({ force: true })
// GOOD - hover to reveal, then click
await tableRow.hover()
await expect(menuButton).toBeVisible()
await menuButton.click()Always import from the custom test utility:
import { test } from '../utils/test.js'Use withFileOnceSetup for expensive setup that should run once per file:
test.beforeAll(async ({ browser, ref }) => {
await withFileOnceSetup(import.meta.url, async () => {
const ctx = await browser.newContext()
const page = await ctx.newPage()
await deleteTestTables(page, ref)
})
})
test.afterAll(async () => {
await releaseFileOnceCleanup(import.meta.url)
})Dismiss toasts before interacting — they can overlay buttons:
const dismissToastsIfAny = async (page: Page) => {
const closeButtons = page.getByRole('button', { name: 'Close toast' })
const count = await closeButtons.count()
for (let i = 0; i < count; i++) {
await closeButtons.nth(i).click()
}
}
await dismissToastsIfAny(page)
await page.getByRole('button', { name: 'New table' }).click()Always include descriptive messages for easier debugging:
// ❌ No context on failure
await expect(page.getByRole('button', { name: 'Save' })).toBeVisible()
// ✅ Clear message on failure
await expect(
page.getByRole('button', { name: 'Save' }),
'Save button should be visible after form is filled'
).toBeVisible()Use explicit timeouts for slow operations:
await expect(
page.getByText(`Table ${tableName} is good to go!`),
'Success toast should be visible after table creation'
).toBeVisible({ timeout: 50000 })Extract reusable operations into domain helpers (e.g. e2e/studio/utils/storage-helpers.ts).
Use the existing wait utilities:
import {
createApiResponseWaiter,
waitForApiResponse,
waitForGridDataToLoad,
waitForTableToLoad,
} from '../utils/wait-for-response.js'Use expectClipboardValue instead of manual clipboard reads with hardcoded timeouts:
// ❌ Brittle
await page.evaluate(() => navigator.clipboard.readText())
await page.waitForTimeout(500)
// ✅ Uses Playwright auto-retries
await expectClipboardValue({ page, value: 'expectedValue' })await page.route('*/**/logs.all*', async (route) => {
await route.fulfill({ body: JSON.stringify(mockAPILogs) })
})Use soft waits for optional API calls:
await waitForApiResponse(page, 'pg-meta', ref, 'optional-endpoint', {
soft: true,
fallbackWaitMs: 1000,
})Clean up test data in beforeAll/beforeEach. Check before deleting to handle existing state gracefully:
const bucketRow = page.getByRole('row').filter({ hasText: bucketName })
if ((await bucketRow.count()) === 0) return
// proceed with deletionReset local storage after tests that modify it:
import { resetLocalStorage } from '../utils/reset-local-storage.js'
await resetLocalStorage(page, ref)cd e2e/studio && pnpm exec playwright show-trace <path-to-trace.zip>cd e2e/studio && pnpm exec playwright show-reportError context files are saved in the test-results/ directory.
Use Playwright MCP tools to inspect UI when debugging locally.
The key difference is cold start vs warm state:
Tests run from a blank database slate. Each test run resets the database and starts fresh containers. Extensions like pg_cron are NOT enabled by default.
pnpm dev:studio-localWhen debugging with a running dev server, the database may already have state from previous runs (extensions enabled, test data present).
Tests that work locally but fail in CI often have assumptions about existing state.
test.describe.configure({ mode: 'serial' }))The test framework automatically resets the database when running pnpm run e2e. This matches CI behavior.
If using pnpm dev:studio-local for Playwright MCP debugging, remember the state differs from CI.
pnpm run e2e -- features/<file>.spec.ts (cold start)test-results/ directorypnpm dev:studio-local and use Playwright MCP tools© supabase, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in .agents/skills/studio-e2e-tests of supabase/supabase.
Open the folder on GitHubat commit 26c838a
Studio E2E Tests next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Studio E2E Tests this skillsupabase/supabase | 111k | — | ~2.8k | Automated safety check: Pass | Apache-2.0 | |
| Hardening Flaky E2E TestsComfy-Org/ComfyUI_frontend | 2.1k | — | ~2.8k | Automated safety check: Pass | GPL-3.0 | |
| Web Application Testinganthropics/skills | 180k | 51 repos | ~966 | Automated safety check: Pass | Apache-2.0 | |
| playwright-cli Browser Automationgithub/gh-aw | 5.4k | 24 repos | ~2.8k | Automated safety check: Pass | MIT | |
| Write and Verify Playwright Testsappsmithorg/appsmith | 41k | — | ~2.9k | Automated safety check: Notes | Apache-2.0 | |
| Cucumber and Playwright E2E Testslanggenius/dify | 158k | — | ~682 | Automated safety check: Pass | Custom licence |
Comfy-Org/ComfyUI_frontend
Diagnoses and fixes flaky Playwright e2e tests by replacing race-prone patterns with retry-safe alternatives.
anthropics/skills
Tests local web applications with Python Playwright scripts, checking frontend behavior, capturing screenshots and reading browser console logs.
github/gh-aw
Drives a real browser from the command line with playwright-cli to open pages, interact, mock requests, save state and work with Playwright tests.
appsmithorg/appsmith
Writes a Playwright end-to-end test from a prompt, runs it against a live Appsmith deployment and retries with fixes up to three times until it passes.
langgenius/dify
Guides changes and reviews of the Cucumber and Playwright end-to-end suite under `e2e/`: feature files, step definitions, support code, tags, locators and assertions.
langflow-ai/langflow
Write and review Playwright E2E tests for Langflow. An agent skill from langflow-ai/langflow.
supabase/supabase
React composition patterns that scale. An agent skill from supabase/supabase.
supabase/supabase
Write, review, and migrate Supabase logs queries against the ClickHouse-backed logs table (the logs.all.otel analytics endpoint).
supabase/supabase
Review Supabase docs changes locally in your supabase/supabase checkout — either an open PR (triage, classify, verify) or your own branch before opening a PR (local self-review).
supabase/supabase
Vitest API and config reference (Jest-compatible) — mocking with vi., spies, fake timers, coverage configuration, fixtures, snapshots, and test filtering.
supabase/supabase
A skill your agent uses whenever code will build, return, fetch, or execute SQL that runs against a user's real Postgres database — even when the request reads like an ordinary feature or bug fix…
supabase/supabase
Error display and troubleshooting pattern for Supabase Studio.
Works with
Categories
Write and run Playwright E2E tests for Supabase Studio (e2e/studio). Studio E2E Tests is an agent skill from supabase/supabase, published by the product's own GitHub organization. Write and run Playwright E2E tests for Supabase Studio (e2e/studio).
Studio E2E Tests fits situations like: asked to run e2e tests; write new E2E tests; failing Playwright tests.
Run `npx skills add supabase/supabase --skill studio-e2e-tests -a claude-code`. Or copy the skill folder (.agents/skills/studio-e2e-tests in supabase/supabase) into .claude/skills/studio-e2e-tests in your project. Claude Code loads it when a task matches its description.
Run `npx skills add supabase/supabase --skill studio-e2e-tests -a codex`. Or copy the skill folder (.agents/skills/studio-e2e-tests in supabase/supabase) into .agents/skills/studio-e2e-tests in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add supabase/supabase --skill studio-e2e-tests -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/studio-e2e-tests, .gemini/skills/studio-e2e-tests, .github/skills/studio-e2e-tests and .opencode/skills/studio-e2e-tests in your project.
Going by SKILL.md and its folder, Studio E2E Tests needs the command-line tools its instructions call (pnpm).
SKILL.md contains no URLs. Any network use would come from the scripts or tools the agent runs. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Studio E2E Tests is published under the Apache-2.0 licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.
About 2.8k tokens (SKILL.md is roughly 11k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Studio E2E Tests: Hardening Flaky E2E Tests (Comfy-Org/ComfyUI_frontend, 2.1k stars), Web Application Testing (anthropics/skills, 180k stars), playwright-cli Browser Automation (github/gh-aw, 5.4k stars) and Write and Verify Playwright Tests (appsmithorg/appsmith, 41k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
supabase (a GitHub organization, an official publisher) maintains it in supabase/supabase, which has 111,222 GitHub stars. The repository holds 22 skills in this directory. The repository was last updated on October 8, 2026.
Source: supabase/supabase on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.