Flowfile Testing And Validation

6. Evidence bar — what "validated" means per change class

A green run is only as strong as what actually executed. Docker-gated suites silently skip without Docker/MinIO/etc — always ask "did the Postgres/MinIO/kernel tests actually run, or did they skip?" (check the pytest summary line for skip counts, not just "passed").

7. xfail / XPASS discipline

Current inventory (verified live 2026-09-12 — re-run before trusting, see §9):

Closed on 2026-09-12 (markers deleted, root causes fixed, branch fix/xfails): the three codegen markers in test_code_generator_edge_cases.py — test_in_operator_numeric (stale XPASS), test_unique_without_columns (engine make_unique now treats columns=[] as all-columns and keeps keep=strategy), test_groupby_with_concat_aggregation (emitter emits str.join(',') via the shared transform_schema.STRING_CONCAT_DELIMITER) — plus the two xfail(strict) scanner evasions in community_nodes/test_security_scan.py (scanner hardened: cross-method self.<attr> decode taint, operator.attrgetter/methodcaller rule; fixtures promoted from evade/ into deny/). The node-designer TestNumericStringAliasBug marker was already gone.

Rule for this repo: XPASS means the xfail is stale. Delete the marker (and the outdated bug description) — never leave it, never "celebrate" the pass. When you add a new xfail for a real known bug, prefer @pytest.mark.xfail(reason=..., strict=True) so a future fix turns into a hard CI failure demanding the marker's removal, instead of a silent XPASS nobody notices.

Verification recipe (the remaining marker lives in the DB-free WASM engine tests):

poetry run pytest \
  "flowfile_wasm/tests/python/test_build_helpers.py::test_filter_advanced_expr_does_not_evaluate_python" \
  -q -p no:cacheprovider -rX
# → "1 xfailed"; an XPASS means the upstream fix landed — raise the pin everywhere and delete the marker

Other skip inventory:

• flowfile_core/tests/flowfile/test_basic_filter.py:751,761 — unconditional pytest.mark.skip, "Manual input converts None to string; test requires actual null values from file sources." Product limitation, not test debt: is_null/is_not_null are effectively untested via manual-input at integration level.
• flowfile_worker/tests/test_train_apply_model.py:159 — benign parametrize carve-out (logistic_regression/knn_classifier need 0/1 targets; covered by a dedicated round-trip test elsewhere).
• No .skip/.todo/.fixme exist in any Playwright or Vitest spec (frontend or wasm) — the TS suites are clean of this pattern.
• The dominant "skip" pattern by volume is environment skipif(not is_docker_available()) across core/worker/frame — dozens of uses. This is a coverage cliff, not debt: on a laptop without Docker+MinIO+emulators running, a large slice of integration surface silently skips, and a green local run is weak evidence of anything Docker-touching.

8. CI test matrix summary (test.yaml)

• Concurrency: group ${{ github.workflow }}-${{ github.ref }}, cancel-in-progress: ${{ github.event_name == 'pull_request' }} — PR runs cancel their own superseded runs; main-branch runs are never cancelled (docker-publish/release pipelines key off completed main builds). This is the only workflow file in the repo with a concurrency block.
• detect-changes (dorny/paths-filter) gates every downstream job by which paths changed; workflow_dispatch input run_all_tests: true forces everything regardless.
• backend-tests matrix: fail-fast: false; ubuntu-latest × Python 3.10/3.11/3.12/3.13, plus macos-latest × 3.11. Starts Postgres/MySQL/MinIO/Azurite/GCS via the poetry run start_* scripts (no-op on macOS CI runners — Docker reports unavailable there). Core runs -m "not kernel". Linux entries then run tests/cloud_e2e -m cloud_e2e after the worker tests, while MinIO is still up (gated on core/worker/shared/tests/cloud_e2e/workflow changes).
• coverage: separate ubuntu/3.12 job, COVERAGE_CORE=sysmon (see §5).
• backend-tests-windows: windows-latest, Python 3.11 only, pwsh shell.
• kernel-tests: ubuntu/3.11, 15-min timeout, builds the kernel image, runs kernel_runtime unit tests then flowfile_core/tests -m kernel.
• check-stubs / check-formula-docs: drift gates — regenerate .pyi stubs / functions.md and fail on git diff.
• test-web: Vitest unit + build:web + preview-server curl check.
• docs-test: mkdocs build.
• test-summary: if: always(), aggregates all jobs except version-sync, fails if any non-skipped job failed. Treat this as the real pass/fail signal for the whole run — but note it is not wired into required branch-protection checks (a CI-mechanics fact, not this skill's territory beyond flagging it).

Separate, path-filtered workflows cover what test.yaml doesn't: e2e-tests.yml (Playwright web E2E; also starts and seeds MinIO, runs core + worker from empty runner.temp dirs with a MinIO-only AWS profile, and fails if either wrote into its working dir), test-docker-auth.yml, test-kernel-integration.yml, test-docker-kernel-e2e.yml, test-kafka-integration.yml, flowfile-wasm-build.yml.

9. History — read this before "fixing" CI test speed again

The backend-tests (ubuntu, 3.12) job was ~56 minutes before 2026-06-20, caused by two compounding factors: coverage's default C-tracer roughly doubling runtime, and the core suite (~5k tests) running fully serially. Shipped fix (merged, do not re-litigate the same options without reading this first):

1. COVERAGE_CORE=sysmon — switched the coverage job to the PEP-669 sys.monitoring tracer, near-zero overhead on 3.12 vs. the old C-tracer's ~2x tax.
2. Coverage split into its own dedicated job — off the critical path of the functional matrix; the plain backend-tests matrix jobs run without --cov at all.
3. Dropped redundant frontend builds from backend jobs.
4. concurrency: cancel-in-progress for PR runs.

pytest-xdist was evaluated and explicitly deferred — do not casually re-propose it. Hard prerequisites that would need to be solved first, all rooted in the same fixed-DB-path problem as §4:

• Each xdist worker needs its own FLOWFILE_DB_PATH, derived from PYTEST_XDIST_WORKER, and that derivation must happen before the first flowfile_core import — the DB engine binds to a URL at import time, so setting the env var after import is a no-op.
• If used on the coverage job: parallel = true in [tool.coverage.run] plus a coverage combine step, neither of which exist today.
• Expected speedup is ~1.4–1.8x, not 2–4x, because several fixtures are shared-service singletons (the worker on :63579, Postgres on :5433, MySQL on :3307) that don't parallelize cleanly across workers without further isolation work.

Without those prerequisites, naive -n auto reproduces exactly the "no such table" cascade from §4 fragile point #1, at coverage-job scale — an empty or garbaged coverage.xml is the typical failure mode.

Load-bearing sleeps — do not remove these thinking they're dead time:

• Catalog test time.sleep(1.05) calls exist because SQLite's updated_at column has 1-second granularity; a faster sleep produces flaky ordering assertions.
• 2-second sleeps around cancel-flow tests are similarly timing-load-bearing.

10. Slow / flaky areas

• Core suite is the long pole (~5k tests, serial) even post-fix; expect the 3.12 matrix job around 28–31 minutes as of the 2026-06-20 fixes.
• Docker image builds dominate -m kernel (mitigate by presetting FLOWFILE_KERNEL_IMAGE to skip the ~30s build) and -m docker_integration (builds core+worker+kernel; conftest uses 600s build timeouts).
• MySQL container start can take up to 60s; first-time image pull up to 300s.
• Worker viz tests are tagged @pytest.mark.slow (flowfile_worker/tests/test_catalog_visualize.py); deselect with -m 'not slow'.
• Playwright: retries: 2 in CI plus trace/video on first retry — this retry budget can mask genuine flakes; workers: 1 avoids port conflicts, not a performance choice.
• A trailing-slash axios/FastAPI mismatch has historically caused silent failures only in Docker (Vite's dev proxy and pytest's TestClient both mask a 307 redirect that Docker's real network path doesn't) — if a route "works locally but not in Docker," check for a slash mismatch in core logs, not test logic.

Provenance and maintenance

Volatile facts below need periodic re-verification — commands are copy-pasteable.

• Pytest markers list (as of 2026-07-03, v0.12.7): grep -A8 '\[tool.pytest.ini_options\]' pyproject.toml
• Coverage config: grep -A20 '\[tool.coverage.run\]' pyproject.toml
• FLOWFILE_DB_PATH / TESTING priority order: sed -n '400,420p' shared/storage_config.py (look for get_database_url)
• Startup-migration skip gate: sed -n '1,30p' flowfile_core/flowfile_core/database/init_db.py
• SKIP_WORKER_TESTS wiring: grep -n "SKIP_WORKER_TESTS\|def flowfile_worker" flowfile_core/tests/conftest.py
• Docker fixture ports: grep -n "_PORT = int(os.environ.get" test_utils/*/fixtures.py
• Test-utils Poetry scripts: grep -n '^start_\|^stop_' pyproject.toml
• xfail inventory + live status (re-run periodically — bugs get fixed and markers go stale silently, that's the whole point of §7; grep -rn "pytest.mark.xfail" --include=*.py . | grep -v "/.claude/" lists every marker):

  poetry run pytest \
    "flowfile_wasm/tests/python/test_build_helpers.py::test_filter_advanced_expr_does_not_evaluate_python" \
    -q -p no:cacheprovider -rX

• Collect counts (~5,079 core / ~311 worker / ~620 frame / ~13 scheduler as of 2026-07-03): poetry run pytest flowfile_core/tests --collect-only -q | tail -3 (repeat per package)
• CI job list, concurrency block, matrix versions: sed -n '1,120p' .github/workflows/test.yaml and grep -n "python-version:" .github/workflows/test.yaml
• Coverage job env/steps: sed -n '216,296p' .github/workflows/test.yaml
• 56-min → ~28-31-min history and xdist deferral: no single file encodes this — cross-check against .github/workflows/test.yaml coverage-job comments (sed -n '216,220p') which corroborate the sysmon rationale; the xdist-deferral reasoning is institutional knowledge captured here, verify by searching git log --oneline --all -- .github/workflows/test.yaml for the speed-fix commit if it needs re-confirming.
• Required branch-protection checks / whether test-summary gates merges: out of this skill's scope — verify via gh api repos/<org>/Flowfile/branches/main/protection if needed for a CI-mechanics task.