Deploying Java SDK Bundles

Deploying Java SDK Bundles

A Java SDK deployment has one artifact: a bundle — your compiled task classes plus the SDK, packaged as a JAR (or a thin JAR alongside its dependency JARs). You build it with Gradle or Maven, then place it in a directory that the JavaCoordinator scans (jars_root) on every worker. This skill is platform-neutral; it shows the build once, then both an open-source and an Astro deployment path.

Experimental. The Java SDK is in preview. Artifact versions below are shown as ${version}; while the SDK is pre-release you may need to build the artifacts into your local Maven repository yourself (see the preview builds section).

Order of operations: build the bundle (this skill) → place it where jars_root points → configure the coordinator (configuring-airflow-language-sdks). The task code itself is authoring-java-sdk-tasks.

Build with Gradle (recommended)

Apply the SDK's Gradle plugin and declare dependencies in build.gradle:

plugins {
    id("org.apache.airflow.sdk") version "${version}"
}

repositories {
    mavenCentral()
}

dependencies {
    annotationProcessor("org.apache.airflow:airflow-sdk-processor:${version}")  // annotation API only
    implementation("org.apache.airflow:airflow-sdk:${version}")
    // Optional logging integration, e.g.:
    // implementation("org.apache.airflow:airflow-sdk-jpl:${version}")
}

airflowBundle {
    mainClass = "com.example.Main"   // your BundleBuilder entry point
    // fatJar = false                // opt out of the single-JAR build (see below)
}

Build it:

./gradlew bundle

The build/bundle/ directory then holds all required JAR(s). Notes:

• The annotationProcessor line is needed only if you use the annotation-based API. The interface-based API doesn't need it.
• By default the plugin produces a fat JAR (via the Shadow plugin) — one self-contained file, which avoids cross-project dependency clashes. Set fatJar = false in airflowBundle for thin JARs; you then deploy every dependency JAR too.
• The Gradle plugin validates that mainClass exists at build time (verifyBundleMainClass).

Build with Maven

Import the BOM so artifact versions and the supervisor schema version are managed in one place:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.apache.airflow</groupId>
      <artifactId>airflow-sdk-bom</artifactId>
      <version>${version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.apache.airflow</groupId>
    <artifactId>airflow-sdk</artifactId>   <!-- version from the BOM -->
  </dependency>
</dependencies>

Wire the annotation processor through maven-compiler-plugin (annotation API only) so it stays off the runtime classpath. Then pick a packaging option:

• Fat JAR (recommended): use maven-shade-plugin. In its ManifestResourceTransformer, set <mainClass> to your BundleBuilder and add the manifest entry Airflow-Supervisor-Schema-Version resolved from the BOM property ${airflow.supervisor.schema.version} (don't hard-code it). mvn package writes the JAR to target/.
• Thin JAR: use maven-jar-plugin to set Main-Class and maven-dependency-plugin (copy-dependencies) to collect runtime JARs into target/bundle/. Here Airflow-Supervisor-Schema-Version is not needed — Airflow reads it from the airflow-sdk JAR on the classpath.

Unlike Gradle, Maven does not validate mainClass at build time; a wrong value only fails at runtime.

Logging integration

For task log records to reach Airflow's log store (and the task log view in the UI), the bundle must include exactly one SDK logging artifact per logging facade you use. Versions are managed by airflow-sdk-bom; Maven users apply the same artifact IDs.

Choosing a facade. For a greenfield project, prefer JPL (System.Logger) — it is built into the JDK, so your tasks need no extra logging API. Pick another facade only when the libraries you integrate with already log through it, so their records reach Airflow too. Preference order: JPL > SLF4J = Log4j 2 > JUL; treat JUL as legacy integration only, not a choice for new code.

Log4j 2 — log4j-core hosts the plugin loader that discovers the appender (log4j-api comes in transitively):

implementation("org.apache.airflow:airflow-sdk-log4j2:${version}")
runtimeOnly("org.apache.logging.log4j:log4j-core:${log4jVersion}")

<Configuration>
  <Appenders>
    <AirflowAppender name="Airflow"/>
  </Appenders>
  <Loggers>
    <Root level="info">
      <AppenderRef ref="Airflow"/>
    </Root>
  </Loggers>
</Configuration>

JUL — call AirflowJulHandler.setup() before any task runs. It clears the root logger's existing handlers (the default ConsoleHandler writes to stderr, which Airflow would otherwise capture as task.stderr at ERROR level, duplicating each record):

public static void main(String[] args) {
    AirflowJulHandler.setup();
    Server.create(args).serve(new MyBundle().build());
}

Don't double up providers. A second System.LoggerFinder implementation alongside airflow-sdk-jpl, or a second SLF4J binding (logback-classic, slf4j-simple) alongside airflow-sdk-slf4j, makes provider selection unpredictable.

Preview builds (before a stable release)

Skip this section if you depend on a stable release. Once you pin a released version (e.g. 1.0.0) published to Maven Central, the mavenCentral() repository in the build snippets above is enough.

While the SDK is pre-release, the documented path is to build the artifacts and the Gradle plugin from the Airflow repo into your local Maven repository:

# in apache/airflow's java-sdk/ directory
./gradlew publishToMavenLocal -PskipSigning=true

Then add mavenLocal() in your project, in both pluginManagement (in settings.gradle) and project repositories (in build.gradle) — this is how the SDK's own example project resolves it.

Once -SNAPSHOT artifacts are published to Apache's snapshot Nexus, that repository can stand in for the local build (same two places):

maven {
    name = "apacheSnapshots"
    url = "https://repository.apache.org/content/repositories/snapshots/"
    mavenContent { snapshotsOnly() }
}

Snapshots move; force a refresh with ./gradlew bundle --refresh-dependencies. (For Maven, add the same repository to <repositories> and <pluginRepositories>.)