Deploying Go SDK Bundles

Deploying Go SDK Bundles

A Go SDK deployment has one artifact: a bundle, a single self-contained native executable that also carries its embedded source and a manifest (the AFBNDL01 format, "the executable is the bundle"). You build and pack it with go, place it where Airflow's ExecutableCoordinator scans, and the Python task runner forks it once per task instance. This skill is platform-neutral: it shows the build, the coordinator wiring, then how to get the bundle onto a worker.

Experimental. The Go SDK is under active development and not production-ready. Everything resolves against the single module github.com/apache/airflow/go-sdk (Go 1.24+).

Order of operations: write the tasks (authoring-go-sdk-tasks) -> build and pack the bundle (this skill) -> place it under executables_root and configure the coordinator -> deploy the matching Python stub DAG.

Build and pack the bundle

The coordinator only recognizes a packed bundle: it scans for the AFBNDL01 trailer and silently skips any file that lacks it, so a plain go build binary is not deployable on its own. Use the packer, shipped as a Go 1.24 tool directive in go.mod (no global install, version pinned per project):

go tool airflow-go-pack ./example/bundle                              # build + pack in one step
go tool airflow-go-pack --goos linux --goarch amd64 ./example/bundle -- -trimpath  # cross-compile; flags after -- pass to `go build`
go tool airflow-go-pack --executable ./bin/sample-dag-bundle --source main.go --airflow-metadata <airflow-metadata.yaml> # pack an existing binary
go tool airflow-go-pack inspect ./bin/sample-dag-bundle               # inspect a packed bundle

The packer builds the binary, execs it with --airflow-metadata to capture the manifest from RegisterDags, then appends source + manifest + a 64-byte trailer. The result is one runnable file.

• Build for the worker's OS/arch. The bundle is a native executable and is not portable; cross-compile with --goos/--goarch. A mismatched binary fails on the worker with exec format error.
• Re-pack after any change to the binary. Re-stripping, re-signing, or swapping in a debug build invalidates the trailer's binary_sha256, and the bundle is then rejected.

Wire up the coordinator

Python's ExecutableCoordinator scans executables_root, matches the incoming dag_id against each bundle's embedded manifest, verifies its integrity hash, then forks the bundle. No Go process runs on the host.

1. Place the packed executable under a scanned directory:

   cp ./bundle /opt/airflow/executable-bundles/   # identified by the AFBNDL01 trailer, not by filename

2. Register ExecutableCoordinator and route the queue to it (see configuring-airflow-language-sdks):

   [sdk]
   coordinators = {"go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}
   queue_to_coordinator = {"golang": "go"}

3. Deploy the matching Python stub DAG; its queue= must equal the queue_to_coordinator key (golang here), and its dag_id/task_ids must match what the bundle registered.

Deployment paths

The SDK runs on any Airflow with the Task SDK; Astronomer tooling is not required.

Docker / Kubernetes

Cross-compile the bundle for the image's platform and bake it in. No Go runtime or worker process is needed in the image; the Python task runner forks the bundle.

FROM apache/airflow:3.3.0        # the language SDKs target Airflow 3.3+
COPY ./executable-bundles/ /opt/airflow/executable-bundles/
# set AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as env vars

On the Helm chart, bake the bundle into a custom image as above or mount it via a shared volume, and set the [sdk] config through environment variables on the worker/scheduler. See deploying-airflow for the broader Docker Compose and Helm workflow.

The apache/airflow:3.3.0 tag above is illustrative: the language SDKs need Airflow 3.3 or newer. Pin whatever current 3.x you actually run rather than copying this tag from memory; read the base image's current tags or docs.