Flowfile Change Control

Flowfile change control

When NOT to use this skill

• Writing or debugging tests → flowfile-testing-and-validation.
• Local dev-server / build / Docker setup → flowfile-build-and-env.
• Env vars and feature flags → flowfile-config-and-flags.
• Cross-package contracts (core/worker/kernel/shared) → flowfile-architecture-contract.
• Digging into a specific past bug for its own sake → flowfile-failure-archaeology.
• Writing/updating docs or CLAUDE.md files → flowfile-docs-and-writing.

This skill is the gate-keeper: what is and isn't allowed to change, how CI proves it, and how a release actually ships.

1. How change is classified and gated here

Every change lands in one (or more) of these lanes. Know your lane before you touch files:

CI is the workflow set under .github/workflows/ (ls it for the current list). The legacy codeql.yaml fails every Monday because it references the missing .github/codeql/codeql-config.yml. This is harmless noise, not a blocker: GitHub Advanced Security default setup is separately configured and live (gh api repos/<org>/Flowfile/code-scanning/default-setup → "state":"configured", weekly, covers python/js-ts/actions/rust) and is what actually reports CodeQL results.

test.yaml (the primary CI gate, ~620 lines) is the only workflow with a concurrency group: PR runs cancel superseded runs of themselves, but main-branch runs are never cancelled — the file's own comment explains why: "docker-publish / release pipelines key off completed main builds."

2. Non-negotiables — with rationale and the incident behind each

2a. Never force-push main

CONTRIBUTING.md: "Don't force-push to main. Releases build from it." docker-publish.yml publishes kernel Docker images from main (kernel-path-filtered, only when the kernel version is unpublished; app images publish from v* tags); test.yaml runs full CI from main. Live branch protection additionally sets allow_force_pushes: false, and repo ruleset id 2660650 ("Only admin commits") adds non_fast_forward + required_linear_history blocks (verified live via gh api repos/.../branches/main/protection and gh api repos/.../rulesets). A force-push to main would desync in-flight release/Docker pipelines that key off a specific commit.

2b. Version bumps move in lockstep — never hand-edit a manifest

Five files carry the app version and must always agree:

1. pyproject.toml → [tool.poetry] version (line 3)
2. shared/_version.py → __version__
3. flowfile_frontend/src-tauri/Cargo.toml → [package] version
4. flowfile_frontend/package.json → "version"
5. flowfile_frontend/src-tauri/tauri.conf.json → "version"

This exists because of a real production incident (commit b21f518c, PR #547, "Centralize version management across all manifests"): the version used to be read from 8 places with drifted hardcoded fallbacks (0.5.0 / 0.12.0 / 0.12.3 / "unknown"). In frozen PyInstaller sidecars, importlib.metadata.version("Flowfile") doesn't resolve, so a fallback could write a non-version string into the NOT NULL db_info.app_version column — breaking desktop startup. The fix built the tooling below; use it, don't reinvent it.

# Bump — rewrites all 5 files, nothing else
python tools/bump_version.py X.Y.Z
make bump-version VERSION=X.Y.Z        # same, via Make

# Check — prints all 5, exits 1 on drift
python3 tools/check_version_sync.py
make check-version                     # same, via Make

# Release-gate form: also assert the canonical version equals a value
python3 tools/check_version_sync.py --expect X.Y.Z

Gotchas:

• make bump-version also refreshes Cargo.lock (cargo update -p flowfile) when cargo is on PATH; calling tools/bump_version.py directly does not, so run that command yourself in that case.
• kernel_runtime/pyproject.toml and flowfile_wasm/package.json (the flowfile-editor npm package) are deliberately NOT synced to the app version — they have their own release cadence (kernel image version, wasm-v* npm tags). Don't "fix" them to match.
• The version-sync CI job runs unconditionally on every push/PR but is not in test-summary's needs list and is not a required branch-protection check — a version-sync failure fails that job but won't by itself block a merge the way you'd expect. Don't rely on it as your only signal; run make check-version yourself before opening a version-touching PR.

2c. Stub drift gate (make check_stubs) — fails CI silently for newcomers

flowfile_frame ships committed .pyi type stubs for its public surface (FlowFrame, Expr, every submodule, py.typed). If you add/change/remove a public method, class, or top-level symbol in flowfile_frame and don't regenerate stubs, check-stubs (CI job, gated on the backend_frame path filter) fails with no obvious link back to what you changed — it just reports a .pyi git diff.

make stubs          # regenerates all .pyi files, then prunes unused imports (ruff --select F401 --fix)
make check_stubs    # stubs + `git diff --exit-code` on the .pyi files; CI runs exactly this

Three generators run in sequence: expr_stub_generator.py → expr.pyi, flow_frame_stub_generator.py → flow_frame.pyi, submodule_stub_generator.py → every other .pyi including __init__.pyi. Always run make stubs after any public API change and stage the diff (agents: hand off the commit per §6) — this is the single most common newcomer-trips-CI moment for anyone working in flowfile_frame. Full stub-authoring guidance lives in flowfile-frame-and-codegen; this skill only covers the gate.

2d. Formula-docs gate (make check_formula_docs) — same failure shape, different surface

docs/users/formulas/functions.md is auto-generated from polars-expr-transformer docstrings (header comment in the file itself says so — never hand-edit it). Bump the polars-expr-transformer pin, or otherwise change what generates that page, without regenerating it, and check-formula-docs (CI job, gated on a formula_docs path filter covering the generator script, the doc file, pyproject.toml, and the lockfile) fails on an unrelated-looking doc diff.

make formula_docs         # regenerate docs/users/formulas/functions.md
make check_formula_docs   # formula_docs + `git diff --exit-code` on that file; CI runs exactly this

CONTRIBUTING.md doesn't mention this gate, so a contributor bumping polars-expr-transformer for an unrelated reason can hit a red documentation.yml build. If you see check-formula-docs fail, this is why; run make formula_docs and stage the regenerated page (agents: hand off the commit per §6).

2e. Alembic migrations: numeric prefix discipline

flowfile_core/flowfile_core/alembic/versions/ holds the numbered chain (001_initial_schema.py onward); ls the directory for the current head — never trust a count in prose.

Rules, with the incident behind each:

• Add a new migration for any change to flowfile_core/flowfile_core/database/models.py; never hand-edit a migration that has already merged to main. Alembic itself was retrofitted in response to a real bug (commit 0ded1ebf, PR #403) after a run-type mismatch between local and Docker databases needed an undocumented downgrade path (006_normalize_run_type.py) — the whole migration system exists because "align local db and worker db" had to be attempted twice by hand before Alembic was added.
• Approved exception — PR #738 only: the maintainer approved dialect-portability edits to revisions 002, 016, 020 and 026 so a fresh PostgreSQL catalog can traverse the existing chain. Limit edits to dialect-specific defaults/SQL and constraint handling; preserve SQLite schema and behavior, revision IDs and ordering. Schema redesign and data-semantic changes still require a new migration. Validate with test_sqlite_schema_unchanged using CATALOG_BASELINE_REF set to the pre-portability base, plus real PostgreSQL upgrade/downgrade coverage.
• Numeric-prefix collisions bite long-running branches. Two branches cut from the same base each add, say, 029_*.py independently; whichever merges second collides or silently shadows revision ordering. This has already happened in-tree (b484a117 fix migrations on a long-lived branch was a direct fix for exactly this). Before adding a migration, check origin/main's current highest number, not your branch's — someone else may have already claimed NNN.
• Importing flowfile_core has a side effect you need to know about: run_startup_migration() fires automatically at import time (flowfile_core/flowfile_core/database/init_db.py) unless FLOWFILE_SKIP_STARTUP_MIGRATION is set. This runs Alembic against whatever DB get_database_url() resolves to — including your live local catalog DB if you're not careful. Set FLOWFILE_SKIP_STARTUP_MIGRATION=1 for any diagnostic import of flowfile_core that isn't meant to touch the DB.

2f. Deliberate dependency pins — do not "fix" these without a mandate

If an agent (or CodeQL, or a linter) flags any of these pins, the correct action is to leave it alone and, if truly necessary, open a Discussion/issue to get the maintainer's sign-off first — not to "fix" it inline.

2g. Deliberate non-features — don't treat NotImplementedError as a TODO

The codebase has a set of intentional, by-design refusals — places where a feature is scoped out, not half-built. Examples: standalone Polars codegen refuses to emit code for external-source / cloud-storage / Kafka / catalog nodes ("Use FlowFrame export" instead — connector_handlers.py, code_generator.py); exported projects raise NotImplementedError for server-backed flowfile_ctx calls (global artifacts, catalog) with warnings surfaced in the export manifest; Kernel (Python Script) nodes are skipped by Export-to-Python; flowfile_frame refuses lambdas in expressions and join_asof/join_where; the bundled flowfile CLI web UI only accepts localhost:63578. If you find one of these while working a task, it is a documented product boundary, not a bug to close — check the surrounding code/docs for the refusal message before "completing" it. Full inventory of these boundaries belongs to flowfile-node-development / flowfile-frame-and-codegen; this skill flags the pattern so you don't file (or fix) a phantom bug.

3. Release mechanics

3a. The checklist

1. make bump-version VERSION=X.Y.Z
2. Only if cargo was missing during the bump: cd flowfile_frontend/src-tauri && cargo update -p flowfile
3. make check-version — must print "All versions in sync"
4. If this release adds a telemetry event or prop (shared/telemetry.py EVENTS changed): redeploy tools/telemetry_collector before tagging — the deployed collector silently drops unknown events, and curl -s https://events.flowfile.app/health | jq .schema must already list them
5. Open a PR (branch protection blocks direct pushes to main), get it merged
6. Tag the merge commit lowercase vX.Y.Z and push the tag
7. This fires pypi-release.yml, release.yaml, and docker-publish.yml (app Docker images) simultaneously (§3b)
8. First release on the tag-triggered path: confirm docker-publish.yml actually fired on the tag (tag pushes ignore the paths: filter — documented GH behavior, but unexercised here). If it didn't, workflow_dispatch with publish_app: true publishes the app images at the manifest version — the intended escape hatch, safe because the new version's tags don't exist on Docker Hub yet.
9. Automatic, but confirm it: the release job generates latest.json with tools/make_latest_json.py and attaches it to the release (§3d) — check the asset landed (gh release view vX.Y.Z --json assets | grep latest.json)

3b. One v* tag push → three workflows

The shared gate means: tag before bumping = all release pipelines fail fast, which is the intended failure mode (better than shipping a mismatched artifact).

wasm-v* tags separately fire npm-publish-wasm.yml, which publishes flowfile-editor to npm via trusted publishing.