Flowfile Failure Archaeology

Flowfile Failure Archaeology

Purpose: this is the "have we been here before" file. Scan the quick index below before writing a fix for something that feels familiar — a race condition, a silent revert, a packaging break, a codegen bug. Several incidents below have a matching "verified and rejected" fix that someone already tried and walked back; re-proposing it wastes a review cycle.

All facts here were re-verified against the working tree / git history on 2026-07-03 at commit f6963c77 (repo version 0.12.7). SHAs are copy-pasteable into git show <sha> or git log -1 <sha>.

When NOT to use this skill

• Need the current non-historical contract for a subsystem → flowfile-architecture-contract.
• Actively debugging a live failure ("has this happened before" is a means, not the goal) → flowfile-debugging-playbook.
• Need build/CI/testing mechanics, not why they're shaped this way → flowfile-build-and-env / flowfile-testing-and-validation.
• Need env var / feature-flag semantics, not the incident behind them → flowfile-config-and-flags.
• Adding a new node type, want current conventions not history → flowfile-node-development.
• Want the open TODO/debt backlog rather than closed incidents → not this file.

Quick index (symptom keyword → incident #)

1 worker hangs forever / no timeout / p.join() / result queue · 2 Kafka/Redpanda node or scheduler sync · 3 app version wrong/blank on Windows, importlib.metadata · 4 SQL viz 504, VizSessionRegistry, response_q · 5 "File must end with PAR1", kernel parquet I/O · 6 kernel won't start / double-starts on fast click · 7 Tauri sidecar spawn/shutdown ordering · 8 Windows/macOS E2E always green · 9 macOS codesign / Python.framework notarization · 10 DMG missing files, huge asar, electron-builder bump · 11 anything still says "Electron" · 12 icons/fonts missing only in the desktop app, not web · 13 output path defaults, "." resolves next to the binary · 14 sort/ node output order wrong only in generated code · 15 flow tab title / display name vs filename · 16 save-as / open overwriting an existing flow · 17 upgrading fastapi · 18 adding/renumbering an Alembic migration · 19 CI job slow/flaky, or want pytest-xdist · 20 file-explorer path traversal / sandbox escape · 21 worker transport (HTTP polling vs WebSocket) · 22 GA4 / any connection failing only when flow is opened · 23 Airbyte, in-house fuzzy match, website/ · 24 draggable panels, z-index, overlay resize · 25 run history / flow-version linkage · 26 polars version bump, silent CI hang/timeout, SQLContext + scan_delta deadlock · a git tag or "since v0.9.4" claim → Tag/release gotchas below · an old/unmerged branch that "already does this" → Branch inventory below · lowering the worker poll, the catalog sleep, coverage parallel=true, re-trying the FastAPI bump → Verified-and-rejected below.

---

Incidents

1. Worker result-queue deadlock (#564) — the flagship. Symptom: worker tasks hang forever, no timeout (wide-table calculate_schema, deep query plans trigger it). Root cause: both transports (spawner.handle_task, streaming.ws_submit) joined the subprocess before reading its result queue — a child put()ing a payload bigger than the OS pipe buffer (~64KB) blocks in its feeder thread until the parent reads, so p.join() first deadlocks parent and child. Fix: flowfile_worker/flowfile_worker/spawner.py:20 def drain_result_queue(q, p, timeout=...) polls and bails the moment the child exits without a result; streaming.py:316 calls it via asyncio.to_thread; apply_model_task now put()s before signalling progress 100 so progress == 100 always implies the result is already queued. Evidence: 3857aced "fix(worker): drain result queue before joining child to avoid deadlock (#564)", 2026-07-02; regression test at flowfile_worker/tests/test_result_queue_deadlock.py pushes a >64KB payload through both paths — self-guarded since pytest-timeout isn't installed here. Status: merged. Rule: never p.join() before draining q; any new task func must put() before signalling progress==100.

2. Kafka pushed straight to main, reverted, re-landed as PRs. Kafka connection management lands via PR 572012e1 (#380, 2026-04-02). Next day two commits go direct to main, no PR: fa86d940 "Adding kafka to flowgraph" (also accidentally committed diff.txt, 4144 lines, and current-review.md, 119 lines, straight to main) and b7e0be80 "Adding kafka sync solutions to the scheduler". Both reverted same day (84392a6c, 3ab65033). Work re-landed properly as 434980b3 (#400) and 7be19166 (#404). Status: shipped (own CI workflow, kafka pytest marker). Watch for stray plan/diff files landing on main — recurred (775fda49, f670c126).

3. Windows app_version hotfix chain → root-cause fix. Three escalating hotfixes to init_db.py:update_db_info in ~36h: ff2644d8 (2026-06-23, PackageNotFoundError fallback → hardcoded "0.12.3") → ebeca9a4 (same day, add bare except Exception too) → 95f60158 (2026-06-24, importlib.metadata.version() was returning None, no exception, in the PyInstaller Windows binary — added explicit is None guard). Root-cause fix same day: b21f518c "Centralize version management across all manifests (#547)" — shared/_version.py (__version__, get_version()) is now the single source of truth, plus tools/bump_version.py / tools/check_version_sync.py (CI drift gate). Rule: never read app version via importlib.metadata in frozen builds — use shared/_version.py::get_version().

4. VizSessionRegistry response-queue race → 504s under SQL viz. 969dd74d (2026-04-26): two parents sharing a session_key both blocked on the same response_q and stole each other's responses; the correct response for parent B got discarded by parent A's "stale, draining" branch, so B timed out at 115s and 504'd. Fix: per-SessionHandle parent_lock: threading.Lock around put-and-wait in execute(); drain loop replaced by a single get() + hard raise on request-id mismatch. Ten minutes later b9dfed1e fixed a companion field-name drift: worker responses use column_schema, catalog/service.py read data["schema"] and silently fell back to a local read via broad except Exception. Rule: shared mp queues need a per-handle lock (pattern in flowfile_worker/flowfile_worker/viz_sessions.py); grep both sides of worker↔core JSON field names when renaming.

5. Parquet corruption race between host and kernel container (#302). Symptom: "File must end with PAR1" reading kernel input/output. Root cause: write_parquet() can leave data in OS buffers; sharing files between host and Docker container over a mounted volume lets the reader see an incomplete file. Fix: explicit fsync after writing parquet in both flowfile_core/flowfile_core/flowfile/flow_graph/ and kernel_runtime/kernel_runtime/flowfile_client.py. Evidence: 84b8e416, 2026-02-03. Rule: any new host↔kernel file handoff must fsync before signalling readiness — Docker volume mounts don't guarantee write visibility ordering.

6. Kernel double-start race. Start button stayed clickable while a kernel was starting; a double-click fired two starts that collided on the container name. Fixed both layers: UI disables the button mid-flight; flowfile_core/kernel/manager.py::start_kernel treats an already-STARTING kernel as a no-op (f43874bb, 2026-05-26). Rule: guard the state machine server-side; never trust the client to disable the button correctly.

7. Tauri sidecar lifecycle races. 161682ba (2026-05-29, 57 insertions, terse message) hardens spawn/shutdown ordering in src-tauri/src/sidecar/{mod.rs,shutdown.rs} and state.rs right after the Electron→Tauri migration (#11). Pairs with the shutdown ladder (HTTP /shutdown → SIGTERM → SIGKILL) and the macOS quit-path fix (Cmd+Q fires RunEvent::Exit not ExitRequested; sidecars leaked until lib.rs matched both plus a parent-death watcher). Status: still the most fragile part of the desktop shell — quit-path desktop E2E is a TODO per the header of flowfile_frontend/tests/web-flow.spec.ts.

8. Windows Electron tests were green while testing nothing (#184). Four compounding anti-patterns made Windows E2E always pass: safety timeout used OR instead of AND across two startup signals; test.skip() instead of throwing on app-launch failure; .catch(() => false) swallowing; an explicit "mark test as passed anyway" branch on unexpected window close. Fix: all four inverted. 09ffc9df, 2026-01-10; follow-up 67e64190 (#187). Status: the suite this hardened was later deleted in the Tauri migration (#11) — fix outlived its own tests. Rule: in Playwright helpers, failures must throw; never test.skip() a broken launch; never force-pass.

9. macOS codesign failure on Python.framework (#266). Electron v36 + electron-builder v26 broke codesign on the PyInstaller-bundled Python.framework ("bundle format is ambiguous") because hardenedRuntime forces --timestamp --options runtime. Fix: signIgnore for Python.framework; CSC_IDENTITY_AUTO_DISCOVERY=false in the CI test workflow. Evidence: 97808864, 2026-01-27.

10. electron-builder v26 DMG breakage + 1GB asar (#269, #271). electron-builder ^26.5.0 → ^25.1.8 (b424fdbf, 2026-01-28): v26.5.0 shipped DMGs missing the Electron Framework binary; also dropped node_modules from the files array (~1GB of dev deps was landing in the asar) and compression "maximum" → "normal" (maximum made asar reads slow). Companion 99c92db1 (#271): vite.config.js → .mjs (ESM-only @vitejs/plugin-react v5 caused ERR_REQUIRE_ESM); pypi-release workflow moved to Node 20. Status: whole Electron toolchain later deleted (#11); lesson generalizes — bundler major bumps repeatedly broke packaging in non-obvious ways.

11. Electron → Tauri migration (#462). 3777c661, 2026-05-30: 100 files, +10,653/−6,621. Deleted the Electron main process (flowfile_frontend/src/main/*.ts) and its whole E2E suite (tests/app.spec.ts, tests/complex-flow.spec.ts, tests/helpers/electronTestHelper.ts). Added src-tauri/, tools/rename_sidecar.py, tools/sign_macos_sidecars.sh. Status: desktop-shell E2E was never re-landed — root CLAUDE.md still calls Tauri-shell E2E via tauri-driver a follow-up, so #8/#9's hardening was discarded with the platform. Some docs pages still say "Electron" — treat any such mention as possibly stale. Aftermath: 8aa1e5c6 (ship .deb only), a812a326 (release matrix), 161682ba (#7), a1e397d1 (#494, overlay), da2a5f2c (#529, see 13).

12. AG Grid icons vanished in Tauri — the CSP trap (#514, #515). Two same-day PRs (2026-06-13) share the literal subject "Fix missing icons in aggrid table viewer" but are unrelated. 3ffa00d0 (#514) is the real fix: src-tauri/tauri.conf.json's CSP lacked font-src 'self' data:, silently blocking AG Grid's icon font only in the desktop webview (fine in npm run dev:web) — current CSP includes it, verify with grep -n csp flowfile_frontend/src-tauri/tauri.conf.json. 4913e27a (#515) reused the title but is actually perf: include_output: bool = True added to FlowNode.get_node_data / GET /node so the settings panel opens instantly instead of computing output-schema prediction. Rule: a desktop-only rendering bug → check tauri.conf.json CSP first; don't trust PR titles here — read the diff.

13. Writing output next to the app binary killed the worker (#529). da2a5f2c, 2026-06-19, issue #526: a . default output path resolved to the app install location; writing there crashed the worker with an opaque error. Fix: default output location is now the user's home directory. Windows/desktop path handling is a repeat offender — see also 082a0136 (#496).

14. Sort codegen silently flipped Descending → ascending (#544). The generator computed descending = (how == "desc"), but visual sort nodes store how as "Descending"/"Ascending" — generated Polars code sorted ascending for a Descending sort, inverting downstream logic with no error raised (the executed graph was correct; only exported code was wrong). Fix: centralized parsing in flowfile_core/flowfile_core/schemas/transform_schema.py:952 def is_descending(how) (+ SortByInput.descending property at line 970), accepting both spellings, used by both codegen (sort + window order_by) and the execution engine, which previously each carried their own copy. Evidence: eac45a01, 2026-06-22. Status: fixed — part of a larger codegen-bug family with the same root pattern (71851cd5 #223, dd521299 #224 join codegen, 65153e0b #512 alias bug, d723e0de #549, 6cd67c4a #553 group_by, a96ab2bf #503). Rule: the visual graph and generated code are two separate executors of the same settings — every codegen change needs an equivalence test against real execution; enums with two spellings need one shared parser, not per-consumer copies.

15. Flow-naming saga — attempted fix, self-revert, current live branch. 2026-04-22, branch claude/fix-flow-tab-display-kM4qD: f060a2e8 "fix: use catalog registration name for flow tab titles", self-reverted the same session (e339ffcb), never merged. As of 2026-07-03, branch improvement/improve-naming-unnamed-flows (unmerged, 1 ahead: fa23a297) would change handler.py: create_flow_name() returns a friendly "Unnamed flow YYYY-MM-DD HH:MM:SS" (main today still returns the old "%Y%m%d_%H_%M_%S_flow.yaml", verified at handler.py:25-27); a new create_unnamed_flow_filename(flow_id) produces Unnamed_flow_<ts>_<flow_id>.yaml for the on-disk path ("registration dedupes on path, not name"). On main today, unnamed flows already live under storage.unnamed_flows_directory — display name and path are already loosely coupled; the branch tightens it further. Status: landed — create_flow_name() returns "Untitled flow <ts>" and create_unnamed_flow_filename(flow_id) gives the unique on-disk path; the branch is gone.

16. Save/open overwrite saga (still in flight). 1133cc77 (#522, 2026-06-16): Save-As handling, SaveDialog, CatalogFlowPicker, +508/−37. 8681308e "Fix open flow overwrite" on branch fix/overwrite-open-flow (2026-07-02, 1 ahead locally): guards in handler.py + new test_flow_dirty_state.py. Origin's copy of that branch has since walked part of it back: b85abc66 "Revert changes to only targeting the catalog" (deleted 81 lines of the dirty-state test, stripped handler/routes changes) and dc9ff9bb "Revert changes in routes.py", both 2026-07-03 — an ambitious dirty-state-tracking approach scoped back to a catalog-only fix. Status: the branch is gone; read handler.py's current save/open guards before writing new logic there.

17. FastAPI upgrade reverted in 2026-05, retried 2026-10. eff7287b "Reverting upgrade Fastapi" (2026-05-11) exists only on branch feature/LLM-security-patches; fastapi = "~0.115.2" stayed from the initial commit until the retry. The upgrade was tried during the LLM-safeguards work (which landed separately as f55aa92c #457) and abandoned before merge; the revert message does not record why. Retried 2026-10-01 (~0.142.2, for the Starlette Dependabot alerts) with the most likely cause fixed: FastAPI 0.132 made strict_content_type=True the default, so a JSON body without a Content-Type header gets a 422 — and ~10 core→worker calls in subprocess_operations.py posted data=model.model_dump_json() with no header. They now pass _JSON_HEADERS. Rule: any new JSON call to a Flowfile FastAPI app must send Content-Type: application/json (or use json=); don't "fix" it by setting strict_content_type=False.

18. Database migrations: born from a production bug. Pre-history: e2977f5c (#422, ~2026-04-08) — a run-type mismatch between local and docker DBs needed a downgrade mechanism, migration 006_normalize_run_type.py, and two tries to align local/worker DBs. Alembic itself arrived in 0ded1ebf (#403, 2026-04-08): alembic.ini, env.py, 001_initial_schema.py, database/migration.py — its own PR body notes "Fix package building with alembic" (bundling it into PyInstaller needed a follow-up fix too). Count/head: ls flowfile_core/flowfile_core/alembic/versions/. Import-time side effect (database/init_db.py:24-27): importing flowfile_core runs run_startup_migration() unless FLOWFILE_SKIP_STARTUP_MIGRATION is set — always set that before importing flowfile_core outside the running app.

19. CI timing & flakiness arc. f8dbb470 "reverting the test order for now" (2025-04-12) — earliest CI-ordering churn, rolled back. Flaky-test firefighting: 66b14004/e3a2bc80 (2025-04-13, select.vue timing), 49e15434 (2025-09-05, deleted 3 lines from test_endpoints.py rather than fix root cause). 90b8dbb2 (#247) — first parallelization. The big one: 093ead72 "ci: split coverage into its own job with sysmon tracer, drop redundant frontend builds, cancel superseded (#539)", 2026-06-20 — coverage now runs in its own job using Python 3.12's sys.monitoring tracer (COVERAGE_CORE=sysmon, test.yaml ~lines 217-264), redundant build:web calls dropped from backend jobs, a concurrency: block (line 21) cancels superseded runs. The coverage job was ~56min before this change; pytest-xdist was explicitly deferred as high-risk — tests share one SQLite DB. See Verified-and-rejected below before re-proposing xdist.