Flowfile Research Frontier

Flowfile Research Frontier

This is a map of unbuilt territory, not a description of shipped features. It names the maintainer's three standing research bets (as of 2026-07-03, v0.12.7), the concrete repo assets each one can stand on, and the discipline for taking a hunch to a merged change. Everything here labeled open or candidate is deliberately not built yet — do not describe it as if it ships.

Use this skill to scope and de-risk exploratory work. It tells you the falsifiable milestone to aim at and the first three PR-sized steps, so a spike produces a measurable result instead of a demo. It does not replace the how-to skills — it points at them.

When NOT to use this skill

• Actually shipping a new node → flowfile-node-development.
• Changing the AI agent/planner/providers → flowfile-ai-subsystem (owns all planner internals).
• Fixing a codegen ↔ graph-execution mismatch → flowfile-codegen-parity-campaign.
• The FlowFrame emission paths / stub pipeline → flowfile-frame-and-codegen.
• Env vars, feature flags, the MutableBool experiment-flag mechanism → flowfile-config-and-flags.
• How to prove a change is correct (test markers, isolation, the evidence bar) → flowfile-testing-and-validation.
• Getting a change merged (PRs, drift gates, review) → flowfile-change-control.
• Why a past idea was abandoned → flowfile-failure-archaeology.

If your task is to build one specific thing, you probably want a sibling above. Come here to decide what is worth building and how you will know it worked.

The three-word compass

The maintainer's frontier, verbatim intent:

• A — AI-native flow building. Agents that reliably build and edit pipelines, not demos.
• B — Visual ↔ code round-trip. Full two-way parity between visual flows, exported Python (raw Polars), and the FlowFrame API.
• C — Decentralized execution on a centralized catalog. Global components, catalog metadata in PostgreSQL, table data in S3 — and every local Flowfile instance brings its own compute, mixing local files and remote tables in complete harmony.

Each pillar below follows the same shape: why the state of the art fails → the specific asset Flowfile already has (verified) → three PR-sized first steps → a falsifiable milestone.

---

Pillar A — AI-native flow building

Why current SOTA fails

AI flow-builders are demos. They generate a plausible pipeline once, on a clean canvas, and fall over on the second turn: they hallucinate column names, loop on tool-name mismatches, and mutate a live graph mid-run so a user's concurrent edit silently clobbers the model's work (or vice-versa). There is no staging, no drift detection, no atomic accept/reject — so "reliable editing of an existing pipeline" (the actual job) is unsolved.

Flowfile's specific existing asset (VERIFIED, as of 2026-07-03 v0.12.7)

Flowfile already has the machinery those demos lack — the diff-staged planner:

• A real multi-turn agent package: flowfile_core/flowfile_core/ai/agents/planner/ (loop.py, insertion.py, recovery.py, staged_schemas.py, …). Not a stub.
• Every LLM tool call goes through execute_tool_call(...) in flowfile_core/flowfile_core/ai/tools/executor/dispatch.py:37, which takes mode: ExecutionMode = "apply". The planner dispatches with mode="stage" so the live graph is never mutated mid-run — steps accumulate as staged entries.
• On completion the staged steps bundle into one reviewable GraphDiff via bundle_staged_results(...) at flowfile_core/flowfile_core/ai/diff.py:318, accepted/rejected atomically in the UI.
• Drift is handled: the session keeps a graph snapshot (sessions.capture_graph_snapshot, re-taken on every resume path in agents/planner/loop.py); a user canvas edit mid-run emits a drift_detected event and parks the session in paused_drift status until an explicit resume.
• Rejected steps are retried by feeding the executor's refusal_detail back as a role="tool" message (ai/tools/executor/refusals.py), and there is a per-session dry-run cache (ai/tools/dry_run.py).

So Flowfile is past the demo layer. The frontier is measured reliability on top of it. (Planner internals are owned by flowfile-ai-subsystem — read it before touching this code.)

First three concrete steps (each PR-sized)

1. Build a held-out benchmark harness. Add a fixture corpus of (natural_language_task, reference_flow) pairs under flowfile_core/tests/ai/, driving agents/planner/loop.py end-to-end against a recorded/replayed provider (the shared LiteLLMProvider seam makes every call interceptable — see flowfile-ai-subsystem). Assert: staged GraphDiff applies cleanly and the applied flow executes to a frame equal to the reference. Report success rate and retry count. This makes "reliable" a number before you touch the agent.
2. Instrument the refusal path. Emit structured refusal-reason counters from ai/tools/executor/refusals.py across the benchmark, so "the agent is flaky" becomes a ranked table of which tool calls fail and why. Failure is free — do not make refusals raise.
3. Add one executor-seam coercion for the top refusal reason, behind an experiment flag (see flowfile-config-and-flags for the MutableBool pattern). Doctrine: when the model consistently emits a recoverable wrong shape, normalize at the executor seam (mirror the existing _coerce_* handlers in ai/tools/executor/coercions.py) — do not keep tightening prompts. Measure the benchmark success delta the coercion buys.

Falsifiable milestone — you have a result when…

Over an N-task held-out benchmark, an agent given a task and a blank-or-partial graph produces a staged diff that (a) applies cleanly and (b) executes to a frame equal to the human reference (assert_frame_equal, order-insensitive), at a measured success rate ≥ your pre-registered threshold, with zero human tool-name corrections and a mean retry count below a pre-registered bound. If the harness cannot report those three numbers, you have a demo, not a result.

---

Pillar B — visual ↔ code round-trip

Why current SOTA fails

Every visual ETL tool is a one-way street. It might export a script, but the export is a lossy, un-reimportable artifact — edit the code and you can never get back to the diagram; edit the diagram and the code silently drifts. There is no tool where the visual graph, hand-written code, and a fluent API are three renderings of one truth that survive a round-trip.

Flowfile's specific existing asset (VERIFIED, as of 2026-07-03 v0.12.7)

Flowfile has both directions of the bridge and one canonical DAG behind them:

• Graph → code: flowfile_core/flowfile_core/flowfile/code_generator/code_generator.py exposes export_flow_to_polars(flow_graph) (line 1602) and export_flow_to_flowframe(flow_graph) (line 1607), via FlowGraphToPolarsConverter / FlowGraphToFlowFrameConverter.
• Code → graph: flowfile_frame builds the same in-process FlowGraph as a side effect of every method call — native nodes where a schema exists, generated Polars-code nodes otherwise (FlowFrame._add_polars_code, flowfile_frame/flowfile_frame/flow_frame.py:588). The graph a frame builds is byte-identical in kind to what the Designer edits.
• Graph → visual editor: open_graph_in_editor(flow_graph) at flowfile/flowfile/api.py:399 saves the frame's graph and opens it in the running Designer.
• The parity discipline already exists: the Descending-sort codegen bug (#544) was root-caused by two executors carrying separate direction parsers; the fix centralized it into transform_schema.is_descending(...) (flowfile_core/flowfile_core/schemas/transform_schema.py:952), used by both the code generator and the execution engine. And there is a real equivalence test: flowfile_core/tests/flowfile/test_code_generator.py parametrizes over [export_flow_to_polars, export_flow_to_flowframe] and asserts the generated code, when executed, yields a frame equal to flow.get_node(N).get_resulting_data().data_frame.

So the pieces exist; what is open is full round-trip idempotence with zero lossy fallbacks. (Emission paths and the codegen bug family are owned by flowfile-frame-and-codegen and flowfile-codegen-parity-campaign — this pillar is the parity campaign's north star.)

The two lossy escape hatches that break round-trip (know your enemy)

1. UnsupportedNodeError in the code generator — a node with no codegen handler.
2. The base64-serialized-LazyFrame fallback in flow_frame.py _add_polars_code: when an expression is not convertible to source (e.g. an unresolvable lambda), the frame serializes a LazyFrame blob into the node and logs a warning ending "a breaking graph when using the the ui" (sic — the doubled "the" is in the source; grep breaking graph when using, flow_frame.py ~line 655). Such a node does not round-trip and the visual graph no longer matches the computation.

Any corpus flow that produces either is, by definition, not round-trippable.

First three concrete steps (each PR-sized)

1. Add a round-trip idempotence test. Extend flowfile_core/tests/flowfile/test_code_generator.py: for a flow corpus, assert export_flow_to_flowframe(g) → re-imported through flowfile_frame → re-exported is idempotent (graph-shape or normalized-string stable), and both executions frame-equal. This surfaces the exact nodes that don't survive the trip.
2. Fail the build on a lossy fallback. Add a test/lint that fails when a corpus flow yields a base64-serialized-LazyFrame node or raises UnsupportedNodeError — converting "sometimes lossy" into a hard, counted gate.
3. Add a native handler for the single most common UnsupportedNodeError node type (in the converter + code_generator/transform_handlers.py), following the one-shared-parser rule (the is_descending pattern) so graph-exec and codegen can never diverge. Ship it with an equivalence test. Cross-ref flowfile-codegen-parity-campaign for the full bug-family playbook.

Falsifiable milestone — you have a result when…

For every flow in a fixed corpus: export_flow_to_flowframe(g) re-imported rebuilds g' such that export_flow_to_flowframe(g') == export_flow_to_flowframe(g) (idempotent), both g and g' execute to assert_frame_equal results, and the corpus produces zero UnsupportedNodeError and zero base64-serialized-LazyFrame nodes. Measurable check: the extended parametrized test is green with those three counters at zero.

---

Pillar C — decentralized execution on a centralized catalog

Why current SOTA fails

Every catalog/lakehouse today assumes central compute: the catalog service owns the engine that reads and writes the data. "Decentralized" offerings still funnel queries through a shared cluster. Nobody offers: shared metadata + shared object-store data, where each participant brings their own local compute and treats a local Parquet file and a remote S3 Delta table as first-class peers in one pipeline, with credentials that never leak between participants.

Flowfile's specific existing assets (VERIFIED, as of 2026-07-03 v0.12.7)

Three seams are already in the right shape:

• Data can already live in object storage, per-catalog. Migration 028 (028_catalog_namespace_storage.py) added storage_uri + storage_connection_name to level-0 (catalog-root) namespaces. flowfile_core/flowfile_core/catalog/storage_backend.py resolve_for_namespace(...) (line 123) resolves a cloud CatalogStorageTarget when a URI is set, resolving the CloudStorageConnection as the catalog owner (owner_id = root.owner_id, line 107) — never the calling user. FLOWFILE_CATALOG_STORAGE_URI / _CONNECTION remain a creation-time default for new catalogs (catalog/services/namespaces.py:_env_default_storage, line 148); resolve_catalog_storage(_user_id, ...) (line 137) is now a shim that ignores its user_id. (Env-var mechanics are owned by flowfile-config-and-flags.)
• The worker reads cloud tables with its own compute. flowfile_worker/catalog_reader.py open_catalog_table(...) (line 29) does pl.scan_delta(validate_catalog_uri(...), storage_options=...) for cloud and local roots alike. Local compute reading remote data is already the code path.
• The catalog-DB seam is a single function. get_database_url() at shared/storage_config.py:402 is the one place every catalog-DB consumer resolves its URL (core database/connection.py, database/migration.py, alembic/env.py, the scheduler's engine.py, shared/run_completion.py). Today it only ever emits sqlite:///….
• Secrets are already instance-independent. A stored secret is $ffsec$1$<user_id>$<fernet_token> — the worker (and any peer) re-derives the per-user key from the shared master key with zero core round-trip (secret_manager.py ↔ flowfile_worker/secrets.py, byte-for-byte parallel). Sharing/decentralizing does not require re-encryption. (This contract is owned by flowfile-architecture-contract.)

Shipped (#738): FLOWFILE_DATABASE_URL/FLOWFILE_DB_PATH accept full SQLAlchemy URLs and _catalog_db_exists is Alembic-based for server URLs. What is open: no two-instance harmony test. The metadata DB is deliberately always-local today; migration 028 is forward-only with no backfill. Also open: global components — user-defined components today are per-instance .py files under <user_data>/user_defined_nodes/ (shared/storage_config.py:109-114, loaded into CUSTOM_NODE_STORE); sharing them through the central catalog is a candidate follow-on to steps 1–3 below, with no design yet.

PostgreSQL catalog support — shipped (#738)

The catalog runs on PostgreSQL 16 via FLOWFILE_DATABASE_URL; test-catalog-databases.yml gates it.