Flowfile Run And Operate

Flowfile Run and Operate

When NOT to use this skill

• Writing/registering a new node type → flowfile-node-development.
• Full .env / feature-flag catalog beyond the handful of ops-critical vars here → flowfile-config-and-flags.
• Secret ciphertext format ($ffsec$1$<user_id>$<token>), HKDF derivation, why core/worker must stay byte-identical → flowfile-architecture-contract (this skill only covers where the master key file lives and how to unblock a RuntimeError at startup).
• Triaging a specific stack trace / reproduced bug → flowfile-debugging-playbook.
• Known historical incidents (the "why" behind a design, postmortems) → flowfile-failure-archaeology.
• Local dev environment setup, make targets, PyInstaller/Tauri build pipeline → flowfile-build-and-env.
• Test-DB isolation (FLOWFILE_DB_PATH for pytest specifically, markers, coverage) → flowfile-testing-and-validation (this skill covers FLOWFILE_DB_PATH for ad-hoc/CLI isolation, which is the same mechanism but a different use case).

If your question is "how do I start/run/inspect a running Flowfile install" — keep reading.

1. The flowfile CLI — every verb and flag

Entry point: Poetry script flowfile = "flowfile.__main__:main" (root pyproject.toml). Argparse lives in flowfile/flowfile/__main__.py.

flowfile [command] [component] [file_path] [--host H] [--port P] [--no-browser]
         [--param KEY=VALUE ...] [--run-id N]

Verb behavior

• flowfile run ui [--no-browser] → flowfile.web.start_server(...). --host/--port are parsed but silently dropped — start_server hard-rejects any host other than 127.0.0.1 or port other than 63578 (raise NotImplementedError). The CLI's own no-args usage text (__main__.py:281) shows flowfile run ui --host 0.0.0.0 --port 8080 as an example — that example is dead code; don't follow it. Use run core/run worker directly if you need a custom host/port.
• flowfile run core → flowfile_core.main.run(host, port), honors --host/--port.
• flowfile run worker → flowfile_worker.main.run(host, port). When launched directly (poetry run flowfile_worker), the worker's own argparse additionally accepts --core-host/--core-port (flowfile_worker/configs.py).
• flowfile run flow <path> [--param k=v] [--run-id N] → headless execution; see §3.A.
• flowfile seed-demo / flowfile remove-demo → flowfile_core.catalog.demo_seed.seed_demo_catalog() / remove_demo_catalog(). Seeds a Demo catalog namespace tree (sales_analytics, market), 4 physical + up to 3 more Delta tables, 2 flow registrations, and 1 cron schedule; prints a summary dict, e.g. {'tables_created': [...], 'sales_flow_registration_id': 1, 'fx_flow_registration_id': 2, 'schedule_id': 1, 'fx_populate': 'triggered'}.
• flowfile project {init|open|save} <folder-or-message> — headless git-backed project tracking (mirrors flows/connections/schedules/catalog into a deterministic git folder; DB stays the source of truth). Owner is get_local_user_id().
  • init <folder>: validates the path (electron: any local root; docker/package: confined to CWD ∪ storage roots, no ..), runs git init, writes a manifest + .gitignore, commits "Initialize Flowfile project". Prints Initialized project '<name>' at <root>.
  • open <folder>: imports flows/connections/schedules from the folder into the DB; prints counts and N value(s) need to be set (FLOWFILE_SECRET_<NAME> or update them in the app). for placeholder secrets.
  • save "<message>": projects the DB into the folder and git commits; prints Saved version <sha8> or (no changes).
• No args → prints FlowFile v<version> + a usage block.

Full multi-line usage/version output, verb list, and demo-seed summary shape are all worth re-reading directly from flowfile/flowfile/__main__.py — read its argparse block; it is the ground truth for every flag above (including the convert/import flags --out, --inspect, --format, --csv, --overwrite).

2. Import side effects (they mutate the live catalog DB)

import flowfile (flowfile/flowfile/__init__.py) unconditionally sets, at import time:

os.environ["FLOWFILE_WORKER_PORT"] = "63578"
os.environ["FLOWFILE_SINGLE_FILE_MODE"] = "1"

So any python -m flowfile ... invocation — including a throwaway python -c "import flowfile" — silently forces single-file (co-hosted worker) mode for the rest of the process.

import flowfile also imports flowfile_core, and flowfile_core/flowfile_core/__init__.py runs, at import time:

1. validate_setup() — node-registry sanity check.
2. init_db() → flowfile_core/flowfile_core/database/init_db.py, which (unless FLOWFILE_SKIP_STARTUP_MIGRATION is set) calls run_startup_migration() — runs Alembic migrations against the live catalog DB — and then seeds the default local_user.

This means importing flowfile or flowfile_core from an ad-hoc script, a REPL, or a debugging session mutates your real catalog database unless you isolate it first.

Isolation rule — use FLOWFILE_DB_PATH for any diagnostic import:

FLOWFILE_DB_PATH=/tmp/scratch/cat.db FLOWFILE_STORAGE_DIR=/tmp/scratch/storage \
  poetry run python -c "import flowfile_core"

Do not combine FLOWFILE_SKIP_STARTUP_MIGRATION=1 with a brand-new/nonexistent DB path — skipping migration means no tables get created at all, and the very next query 500s with "no such table." Only use FLOWFILE_SKIP_STARTUP_MIGRATION=1 when you deliberately want to import core against an already-migrated DB without re-running Alembic (e.g. the Alembic CLI itself, which needs to import settings without triggering a second migration run).

Also: flowfile/__init__.py mutes the PipelineHandler logger to WARNING on import.

3. Headless flow execution — three equivalent paths

All three run the same logic: force execution_location = "local", disable worker offload (OFFLOAD_TO_WORKER.set(False) — compute happens in-process, no worker service needed), delete any UI-only explore_data nodes (prints Skipping N explore_data node(s) (UI-only)), stamp catalog producer lineage, call flow.run_graph(), and report per-node results.

A. CLI (dev / pip install)

flowfile run flow /abs/path/pipeline.yaml
# or, with parameter overrides and a pre-created run-id:
flowfile run flow flow.yaml --param input_dir=/data --param threshold=100 --run-id 42
# equivalently:
python -m flowfile run flow /abs/path/pipeline.yaml

Implemented in flowfile/flowfile/__main__.py:run_flow(). --param key=value (repeatable) replaces an existing FlowParameter's default_value or appends a new one. Exit 0 + Flow completed successfully in X.XXs / Nodes completed: n/m on success; exit 1 + per-node errors on stderr on failure. Writes a per-flow log to <storage>/logs/flow_<flow_id>.log (see §9).

B. PyInstaller / frozen-sidecar binary

python -m flowfile_core.main --run-flow <path> --run-id <id>   # --run-id is REQUIRED here

flowfile_core/main.py dispatches --run-flow (only reachable via if __name__ == "__main__", checked at the top of the file before any router import or the storage sweep) to flowfile_core/run_flow_cli.py:run_flow_cli, which duplicates run_flow()'s logic inside flowfile_core because the top-level flowfile package isn't bundled into the PyInstaller binary. Missing --run-id prints Error: --run-id is required and exits 1 — unlike path A, there is no optional-run-id fallback here.

C. Scheduler-spawned

shared/subprocess_utils.py:spawn_flow_subprocess(flow_path, run_id):

• frozen: [sys.executable, --run-flow, path, --run-id, N]
• dev: [sys.executable, -m, flowfile, run, flow, path, --run-id, N]
• stdout+stderr redirect to shared.run_logs.run_log_path(run_id) = storage.logs_directory / "scheduled_run_<run_id>.log" (mode 0644, truncated each run) — resolved per call, so it honors FLOWFILE_STORAGE_DIR, docker mode, and TESTING. The scheduled_run_ prefix is legacy: manual and on-demand runs use it too.
• start_new_session=True (fire-and-forget); returns the child PID or None on spawn failure.

The scheduler engine (flowfile_scheduler/flowfile_scheduler/engine.py) polls the shared SQLite catalog DB every DEFAULT_POLL_INTERVAL = 30 seconds, supports cron/interval/table-trigger schedules, skips launching a schedule that already has an active run (ended_at IS NULL), creates the FlowRun row (run_type="scheduled") before spawning, and records the child PID. Standalone: poetry run flowfile_scheduler (continuous) or poetry run flowfile_scheduler --once (single tick). Embedded inside core only when FLOWFILE_SCHEDULER_ENABLED is truthy (true/1/yes).

4. Service startup — order, terminals, co-hosting

Local dev — three terminals

poetry run flowfile_core                     # :63578 — start FIRST
poetry run flowfile_worker                   # :63579 — needed for worker-offloaded / remote-execution runs
cd flowfile_frontend && npm run dev:web       # :8080, proxies /api -> :63578

Core must be up before the frontend — Vite proxies /api to core (and nginx does the same in Docker: proxy_pass http://flowfile-core:63578/). The worker is independent: core starts fine without it, but any worker-offloaded node run fails until the worker is up. The worker only calls back to core to ship logs (POST /raw_logs).

Single-process ("unified") mode — the pip-install story

flowfile run ui [--no-browser]

Fixed at 127.0.0.1:63578 (see §1). start_server sets FLOWFILE_MODE=electron if unset, forces OFFLOAD_TO_WORKER.value = True, and extends the core FastAPI app:

• serves the built Vue SPA from flowfile/web/static/ under /ui (missing build → {"error": "Web UI not installed..."}),
• adds StripApiPrefixMiddleware (strips a leading /api so the Docker-oriented frontend base URL still resolves),
• GET /single_mode reports whether FLOWFILE_SINGLE_FILE_MODE == "1",
• mounts the worker's own router at /worker on the same app — one process serves UI, core API, and worker compute.

FLOWFILE_SINGLE_FILE_MODE=1 + FLOWFILE_WORKER_PORT=63578 is exactly what import flowfile sets automatically (§2) — that's why get_default_worker_url() appends /worker to the worker URL when single-file mode is on: core's own offload POSTs go to http://127.0.0.1:63578/worker/..., i.e. itself. If you need this co-hosted mode without the flowfile CLI wrapper (e.g. scripting it directly), set both env vars yourself before starting core.

Gotcha: the browser tab opens after time.sleep(5) but before uvicorn starts listening — the first load can 404/connection-refuse; just reload. webbrowser.open_new_tab targets /ui; API docs live at /docs.

Core startup/shutdown side effects worth knowing

• Import-time: storage.cleanup_directories() runs (see §8 cleanup policy) — starting core deletes cache files older than 1 hour, every time. The --run-flow verb dispatches above it, so headless run children never sweep (flowfile run flow never imports main at all).
• Lifespan startup: logging.basicConfig(INFO, ...) to stderr, where the import-time PipelineHandler console handler (configs/__init__.py) also writes, keeping stdout clean for script output (no file handler — Tauri pipes both streams); starts the embedded scheduler iff FLOWFILE_SCHEDULER_ENABLED.
• Lifespan startup also runs shared.run_logs.cleanup_old_logs() — age-based retention over scheduled_run_*.log and flow_*.log (FLOWFILE_RUN_LOG_RETENTION_DAYS, default 30, 0 disables).
• Lifespan shutdown: stops the scheduler, stops every Docker kernel container, and shuts down the optional local LLM. It does not delete logs; logs expire only by age.
• POST /shutdown triggers a graceful uvicorn exit (used by the Tauri shutdown ladder).
• CLI arg parsing (--host/--port/--worker-port) happens at import of flowfile_core.configs.settings via parse_known_args() against whatever sys.argv the importing process has — importing core inside a process with unrelated --host/--port flags on argv will silently repoint the server.

Docker — build-from-source stack

First-time setup (.env bootstrapping, kernel-image build profiles + FLOWFILE_KERNEL_IMAGE, the Docker-socket mount and fixed-name flowfile-network rationale) → flowfile-build-and-env §2(d); this section only covers operating an already-built stack.

docker compose up -d starts frontend :8080, core :63578, worker :63579. Verified in docker-compose.yml: core and worker both get shm_size: 2gb, and FLOWFILE_SCHEDULER_ENABLED=true + FLOWFILE_ENABLE_PROJECTS=true by default. Core/worker Dockerfiles: CMD python -m flowfile_core.main / python -m flowfile_worker.main, healthcheck curl -f http://localhost:6357{8,9}/docs. Status/logs: docker compose ps, docker compose logs -f flowfile-core|flowfile-worker (§11 runbook).

Published-images deployment ("docker-remote" — documented drift)

There is no docker-remote/ directory. The actual published-images deployment story lives in docs/users/deployment/docker.md: a sample compose file pulling edwardvaneechoud/flowfile-{frontend,core,worker}:latest plus versioned kernel images (edwardvaneechoud/flowfile-kernel-base:0.3.0, -ml:0.3.0), same env vars/volumes as the source compose (the storage volume is just named flowfile-storage there instead of flowfile-internal-storage). Ops commands: docker compose up -d | down | pull | logs -f. Treat any reference to docker-remote/ elsewhere in the docs as stale — point people at docs/users/deployment/docker.md instead.

5. Flow file format

Format: YAML (preferred) or JSON, root Pydantic model FlowfileData. Key shape:

flowfile_version: 0.12.7        # stamped with the running app version at save time; informational only
flowfile_id: 424242
flowfile_name: my_flow
flowfile_settings:               # description, execution_mode (Development|Performance),
                                  # execution_location (local|remote|...), auto_save,
                                  # show_detailed_progress, max_parallel_workers, parameters[]
nodes:
  - id: 1
    type: manual_input           # node type == FlowGraph method suffix ("add_" + type)
    is_start_node: true          # load-order hint only; start nodes are re-derived on load
    x_position: 100
    y_position: 100
    outputs: [2]                 # downstream ids, sorted by (target, handle); read for handle lookup
    output_handles: [output-0]   # parallel to outputs; missing entries default to "output-0"
    setting_input: {...}         # node-type-specific settings
  - id: 2
    type: filter
    input_ids: [1]               # edges are rebuilt from each target's input_ids / left_input_id /
    left_input_id: null          # right_input_id, in saved order (keyed run_flow edges: input_connections)
    right_input_id: null
groups: []                       # visual FlowfileGroup boxes

Save (FlowGraph.save_flow): .yaml/.yml → yaml.dump(..., default_flow_style=False, sort_keys=False, allow_unicode=True); .json → json.dump(..., indent=2); .flowfile raises DeprecationWarning ("The .flowfile format is deprecated. Please use .yaml or .json formats."); unknown extension → warns and defaults to YAML. Save also re-derives the flow name from the filename stem, prunes empty visual groups, and records catalog read-links.

Load (open_flow): dispatches on suffix. .yaml/.yml/.json → straightforward Pydantic parse. .flowfile → legacy pickle load via a custom LegacyUnpickler that maps old dataclass names through tools/migrate/legacy_schemas.py's LEGACY_CLASS_MAP, plus a compatibility pass that back-fills fields legacy pickles lack (groups, output_handles, table_settings). There is no version-gated migration keyed on flowfile_version — compatibility is purely structural (Pydantic defaults + the legacy-pickle path); the stamped version is cosmetic.

Load overrides the flow's name from the filename stem — flow_settings.name = flow_path.stem. Rename the YAML on disk and the flow's in-app name changes on next open; the flowfile_name field inside the file becomes stale/cosmetic.

Path sandboxing at load: allowed extensions {.yaml, .yml, .json, .flowfile}. In docker mode, the validator rejects a relative path that resolves outside flows_directory/uploads_directory/temp_directory_for_flows — but an absolute path outside those directories is not rejected by this check (the guard is if not is_safe and not path.is_absolute(): raise). Don't rely on this function alone as a security boundary for absolute paths; verify current behavior in flowfile_core/flowfile_core/flowfile/manage/io_flowfile.py::_validate_flow_path before treating it as a hard confinement guarantee. Local/electron/package modes accept any path.

Where flows live on disk