Ffi Capsule Protocol

<!---
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing,
  software distributed under the License is distributed on an
  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  KIND, either express or implied.  See the License for the
  specific language governing permissions and limitations
  under the License.
-->

FFI Capsule Protocol

datafusion-python shares Rust objects with extension libraries through PyCapsules. Every hook is a dunder method named __datafusion_<thing>__ that returns a capsule wrapping an FFI-safe struct. They are one protocol, not a collection of unrelated methods, and they have a settled convention that has already been migrated once (see docs/source/user-guide/upgrade-guides.md, DataFusion 52.0.0 and 55.0.0).

Rule 1 — enumerate the family before you change a member

Do this first, every time. It takes one command and it is the whole point of this skill:

grep -rn "__datafusion_[a-z_]*__" --include="*.rs" crates/ examples/*/src/

Compare the signature you are about to write against what the others already do. If yours is shaped differently, that is a finding about your design, not about theirs.

Rule 2 — a getter takes the session it is being installed on

fn __datafusion_physical_extension_codec__<'py>(
    &self,
    py: Python<'py>,
    session: Bound<'py, PyAny>,
) -> PyResult<Bound<'py, PyCapsule>> { ... }

The host calls the getter and passes itself. That argument is how an extension library reaches things only the session has.

SessionContext implements the same getters and ignores the argument, so a session satisfies the protocol too — ctx.__datafusion_query_planner__() and ctx.__datafusion_query_planner__(ctx) are both valid.

__datafusion_session_planner__(ctx, fallback) is the exception to the shape above: it takes a second argument, the planner assembled so far. A session has one planner slot, so planners compose by nesting rather than by chaining, and the host hands each bundle the previous layer instead of letting it capture one. Wrap fallback and delegate to it; returning a planner that ignores it discards every layer beneath, including one the session already had. It runs after every bundle's codecs are installed, so ctx carries the final chains.

That is also the only hook where it does. __datafusion_session_components__ runs before anything is installed, so its ctx still carries the chains the receiver had — the same session, and the same task-context provider, but not this call's codecs, not even your own. Read the host's codec chains in the planner hook, never in the extension hook.

A codec must always be handed over as an object implementing its getter, never as the bare capsule the getter returns; with_extensions refuses a capsule. A codec's wire id — the string a payload names on decode, which has to mean the same thing in whichever process decodes — is read off the object it arrives as, and a capsule has no type to read one from. Deriving the id from the bundle that contributed the capsule is not the fix: the bundle is whatever object the caller passed, so an application packaging your library inside a bundle of its own would re-tag your payloads and they would stop decoding where they are read. If the object's class name is not the identity you want on the wire, declare __datafusion_codec_id__ on it. BundledLogicalCodec in examples/datafusion-ffi-query-planner-example/src/extension.rs is the shape. This applies only to codecs — a query planner carries no wire id.

Rule 3 — never construct a SessionContext in an extension library

The FFI constructors ask for things a library does not have:

Arc::new(SessionContext::new()) is the wrong answer to all three, for two independent reasons:

1. It is the wrong registry. Decode callbacks resolve names against whatever provider the codec carries. An empty session resolves nothing, so a function the host registered with register_udf is invisible to a node that references it by name.
2. It dangles. FFI_TaskContextProvider downgrades its provider to a Weak. A context built inline in the getter is dropped before the capsule is ever used, and every callback then fails with TaskContextProvider went out of scope over FFI boundary.

Prefer the *_with_ffi_codec(s) constructors when they exist. They take prebuilt codecs that already carry the host's provider, so there is no provider parameter to get wrong.

Rule 4 — the helpers live in crates/util/src/lib.rs

ffi_logical_codec_from_pycapsule, ffi_physical_codec_from_pycapsule, ffi_query_planner_from_pycapsule, ffi_task_context_provider_from_pycapsule, table_provider_from_pycapsule. Each takes the object and, where relevant, an Option<&Bound<PyAny>> session:

• Some(session) — importing a foreign object; the getter needs the session.
• None — the object already is a session and is being asked for what it holds.

Adding a getter means adding a helper here, not hand-rolling capsule extraction at the call site.

Rule 5 — changing a getter's signature is a breaking change

Extension libraries implement these methods. A signature change breaks every one of them, and the failure is a bare TypeError from a call1. So:

• Add a section to docs/source/user-guide/upgrade-guides.md with before/after Rust, matching the 52.0.0 and 55.0.0 entries.
• Add the api change label to the PR.
• Map the TypeError to a diagnosable message. call_capsule_getter in crates/util/src/lib.rs already does this; reuse it.
• Update python/datafusion/context.py and python/datafusion/user_defined.py, where the Protocol type hints for these methods live.

Changing what a codec puts on the wire is equally breaking, and easier to miss because no signature moves and nothing fails to compile. Serialized plans outlive the process that wrote them, so the same checklist applies: upgrade guide, api change label, and a statement of exactly which sessions produce different bytes.