Flowfile Debugging Playbook

Deep dive: kernel container triage

Kernel containers back Python Script nodes (sandboxed user code). Each is a Docker container that EXPOSEs port 9999 internally; core maps that to a host port in 19000-19999 and the kernel calls back to core on :63578.

docker ps | grep flowfile-kernel-          # is it even running?
docker logs flowfile-kernel-<kernel_id>    # container-side error?

Container naming is flowfile-kernel-<kernel_id> (the kernel's DB id/uuid, not the flow id). If a stale container from a previous test run or crashed session is holding a port, force-remove it and let core recreate it:

docker rm -f flowfile-kernel-<kernel_id>

Local (non-Docker-in-Docker) mode bind-mounts the cache and catalog-tables directories into the container — if kernel code can't see catalog tables or shared artifacts, verify those directories weren't relocated out from under the kernel-visible volume (shared_directory/artifact_staging_directory/ global_artifacts_directory must stay under <base>/temp/kernel_shared or $FLOWFILE_SHARED_DIR).

Core also stops every kernel container on its own shutdown — if kernels vanish right when you stop core in dev, that's expected, not a leak.

Deep dive: desktop (Tauri) sidecar leaks

The desktop shell spawns flowfile_core and flowfile_worker as sidecar subprocesses and is supposed to reap them on quit via an HTTP /shutdown → SIGTERM → SIGKILL ladder. Two things make this fragile enough to warrant a manual check after any change near src-tauri/src/lib.rs or src-tauri/src/sidecar/:

1. On macOS, Cmd+Q and dock-quit route through RunEvent::Exit, not RunEvent::ExitRequested — code that only handles the latter leaks sidecars on those quit paths specifically (window-close-button quit is unaffected). Always test both quit paths, not just one.
2. A quit that lands mid-startup, or a sidecar that dies exactly during shutdown, can leave a PID uncleared that a later kill targets after the OS has recycled it onto an unrelated process.

There is currently no automated E2E test covering desktop sidecar lifecycle — manual verification is the only guardrail:

# after quitting via the window's close button:
ps aux | grep flowfile_core; ps aux | grep flowfile_worker
# after quitting via Cmd+Q / dock-quit — check again, separately:
ps aux | grep flowfile_core; ps aux | grep flowfile_worker

Any hits after a clean quit mean the shutdown ladder didn't reach that process; kill it manually and treat the reproduction steps as a bug report against the sidecar lifecycle code, not a one-off fluke.

Deep dive: AI agent misbehavior

When an agent run does something surprising (tool-name loops, planner self-loops, references to columns that don't exist, refuses to stop), the fastest path to ground truth is the prompt log — it captures the exact system prompt, full message history, tool catalog, and model response for every LLM call, regardless of provider (all vendors route through the same LiteLLMProvider seam).

export FLOWFILE_AI_LOG_PROMPTS=true     # accepts true/1/yes/on
# reproduce the failing flow/chat/agent session, then:
python -m flowfile_core.ai.prompt_log tail 20
python -m flowfile_core.ai.prompt_log grep '<pattern>' [SURFACE]

The file lives at <storage base>/ai_prompts/YYYY-MM-DD.jsonl (UTC-dated — ~/.flowfile/ai_prompts/... by default), one JSON line per LLM call, jq-parseable. Lines over 256 KiB (runaway loops) keep the system prompt and most-recent turns verbatim and stub older message bodies with [...truncated, len=N chars] plus "truncated": true, so it stays parseable even for pathological sessions. Set FLOWFILE_AI_LOG_PROMPTS_SCRUB=true before sharing a transcript externally — it masks PII in user/tool messages while leaving system+assistant content verbatim (that's the part you're debugging). Both flags default off; don't leave prompt logging on in production.

For the agent architecture itself (assist/copilot/planner surfaces, provider registry, rate-limit scheduler caveats) see flowfile-ai-subsystem.

Where every log file lives

<storage base> = FLOWFILE_STORAGE_DIR env if set, else ~/.flowfile locally, else /app/internal_storage in Docker mode.

Discriminating-experiment discipline

Before you conclude "my change broke N tests" (or "my change fixed the bug"), run the elimination checklist — false positives from shared, stateful infrastructure are the norm here, not the exception:

1. Check for concurrent sessions first. ps aux | grep pytest (another pytest run can drop your shared test DB's tables mid-run — see the "no such table" deep dive above). Also check for a second flowfile_core/ flowfile_worker already listening on the default ports (lsof -i :63578, lsof -i :63579) — your "new" server might be talking through a stale one.
2. Re-run the exact failing subset in isolation, with a scratch DB and without prior state:

   FLOWFILE_DB_PATH=/tmp/ff_recheck_$$.db poetry run pytest \
     path/to/test_file.py::TestClass::test_name -q -p no:cacheprovider -rX

   If it passes alone but failed in the full run, you likely hit shared-state bleed (catalog namespace rows wiped by another module's cleanup, a stale worker/postgres/mysql session-singleton left dirty by a previous run, process-wide env mutation not restored) — not a regression from your change.
3. Compare skip counts, not just pass/fail counts. A large slice of the integration surface only runs when Docker/MinIO/GCS/Azurite/Redpanda are up; a run with more skips than your baseline is weaker evidence, not stronger, even if it's "green."
4. XPASS on an xfail marker means the bug was already fixed — delete the marker, don't add work to preserve it. (Verify with -rX and look for xpassed in the summary line before trusting an xfail's reason text.)
5. Only after 1-4 clear the bar should you attribute a result to your change.

Provenance and maintenance

Volatile facts below are dated 2026-07-03 (v0.12.7) — re-verify with the commands shown before relying on the numbers in a new session.

• App version: grep '^version' pyproject.toml.
• Alembic head: ls flowfile_core/flowfile_core/alembic/versions/ | sort | tail -3, compared against the live DB. Live DB check: sqlite3 ~/.flowfile/database/flowfile_catalog.db 'select * from alembic_version;'.
• results_exists worker-down behavior: grep -n "def results_exists" -A 15 flowfile_core/flowfile_core/flowfile/flow_data_engine/subprocess_operations/subprocess_operations.py.
• invalidate_cache escape hatch: grep -n "def invalidate_cache" -A 10 flowfile_core/flowfile_core/flowfile/flow_node/flow_node.py.
• "Flow no longer in memory" 404 message/route: grep -n "no longer in memory" flowfile_core/flowfile_core/routes/routes.py.
• CORS-masked-500 guard comment: grep -n "CORS" flowfile_core/flowfile_core/flowfile/flow_graph.py.
• Kernel host port range: grep -n "_BASE_PORT\|_PORT_RANGE" flowfile_core/flowfile_core/kernel/manager.py.
• Kernel container naming: grep -n 'f"flowfile-kernel-' flowfile_core/flowfile_core/kernel/manager.py.
• AI prompt log CLI: python -m flowfile_core.ai.prompt_log tail 5 (run against a scratch FLOWFILE_DB_PATH/FLOWFILE_STORAGE_DIR first if you don't want to touch real data).
• Run-log path + retention: grep -n "run_log_path\|cleanup_old_logs" shared/run_logs.py shared/subprocess_utils.py flowfile_core/flowfile_core/main.py flowfile_scheduler/flowfile_scheduler/engine.py.
• Test DB isolation model / shared-fixed-path behavior: grep -n "get_database_url" -A 20 shared/storage_config.py.
• FLOWFILE_SKIP_STARTUP_MIGRATION gate: grep -n "FLOWFILE_SKIP_STARTUP_MIGRATION" flowfile_core/flowfile_core/database/init_db.py.
• Tauri Cmd+Q / RunEvent::Exit handling and the two open TODO races: grep -n "RunEvent::Exit\|TODO(D)\|TODO(C)" flowfile_frontend/src-tauri/src/lib.rs flowfile_frontend/src-tauri/src/sidecar/mod.rs.
• Sidecar process names for ps aux triage: grep -n 'BINARIES = ' tools/rename_sidecar.py (binaries are named flowfile_core/flowfile_worker).
• Trailing-slash 307 pattern: grep -n "proxy_redirect" flowfile_frontend/nginx.conf; grep -n "configure:" flowfile_frontend/vite.config.mjs.