C Extensions With Bocpy

Writing C extensions with the bocpy public C ABI

This skill is for writing a downstream C extension whose custom types must travel through bocpy's runtime — placed inside a Cown, scheduled with @when, or shipped via send/receive across worker sub-interpreters.

Read thinking-in-boc first. The C ABI does not change the BOC mental model. Your C type is still wrapped in a Cown, behaviors are still scheduled with @when, and ordering still comes from the cown graph — not from anything you do in C. The C ABI only buys you a safe, zero-copy way to cross the interpreter boundary; the choreography between behaviors is unchanged.

When you need this skill

You need a C extension that uses the bocpy public C ABI only when all three of the following are true:

1. Your type wraps a native C resource (large buffer, matrix, GPU handle, file descriptor, FFI pointer, etc.) that you do not want to copy or pickle every time it crosses interpreter boundaries.
2. You want instances of that type to live inside a Cown and be acquired by @when behaviors on worker sub-interpreters.
3. You want isolation guarantees: at most one interpreter may read or write the resource at a time, and any stale wrapper left behind in the previous owner cannot observe the resource any more.

If your type is pure-Python, pickleable, or you are happy with a copy on each interpreter crossing, you do not need this skill — Cown(my_obj) already works. Read the BOC primer in the project copilot instructions and stop here.

The canonical worked example shipped by bocpy lives in templates/c_abi_consumer/ of the bocpy source tree — copy it as the starting point for a new extension. The Matrix type in src/bocpy/_math.c is the in-tree reference implementation.

The proto-Region mental model

A bocpy-aware C type implements proto-Region ownership: a single atomic owner field on the impl identifies the interpreter that may read or write the payload. The producer-side XIData callback CASes that field from bocpy_interpid() to BOCPY_NO_OWNER; the consumer-side callback CASes it back from BOCPY_NO_OWNER to bocpy_interpid(). Any data accessor that reads or writes the payload first asserts bocpy_interpid() == atomic_load(&impl->owner) and raises RuntimeError otherwise.

This is not the full Lungfish region model — there are no nested regions, no freeze, no merge, no borrow tracking — but it is enough to turn "a shareable pointer" into "a resource owned by exactly one interpreter at a time", which is what BOC needs.

What the proto-Region contract gives you in practice:

• A stale wrapper held in a producer interpreter after the handoff cannot read or write the impl — every accessor raises RuntimeError until ownership returns. This catches races that pointer-only sharing would miss.
• The CAS in the producer callback fails if the calling interpreter is not the current owner. A behavior that tries to send a cown's value somewhere it should not go is rejected at the boundary, not silently corrupted.

Required scaffolding

setup.py

Use bocpy.get_include() for the header search path and bocpy.get_sources() for the MSVC out-of-line atomics shim (no-op elsewhere). Copy this verbatim from templates/c_abi_consumer/setup.py and change the module name:

from setuptools import Extension, setup
import bocpy

setup(
    ext_modules=[
        Extension(
            "_your_extension",
            sources=["src/_your_extension.c"] + bocpy.get_sources(),
            include_dirs=[bocpy.get_include()],
        ),
    ],
)

pyproject.toml

Declare bocpy in both [build-system].requires (so an isolated PEP 517 build can satisfy import bocpy in setup.py) and [project].dependencies (so installing your wheel installs bocpy at runtime).

Always install with pip install --no-build-isolation so the build resolves headers against the bocpy install actually being tested, not whatever PyPI happens to publish.

Header include

#include <bocpy/bocpy.h>

/* Compile-time guard: bocpy.h must not leak Py_BUILD_CORE. */
#ifdef Py_BUILD_CORE
#error "Py_BUILD_CORE leaked from bocpy.h"
#endif

Rules:

• <bocpy/bocpy.h> includes <Python.h> internally. It is order-insensitive with respect to <Python.h> itself.
• It must appear before any system header (<stdio.h>, <string.h>, ...) in the same translation unit, the same way <Python.h> must — CPython forbids system headers before Python.h.
• C only. <bocpy/bocpy.h> is not supported from C++ in this release. C++ consumers must wrap the ABI in a thin C translation unit.

Public ABI surface — what you may use

Everything below is exposed by <bocpy/bocpy.h>. Treat anything else under the bocpy package directory as private.

Internal headers and surfaces (boc_compat.h, boc_cown.h, boc_sched.h, boc_tags.h, boc_terminator.h, boc_noticeboard.h, the typed atomics, the BOC mutex/condvar types, boc_yield, boc_now_*, boc_sleep_ns, etc.) are not public. Do not depend on them.

The lifecycle: a Counter-sized walkthrough

The full annotated source is templates/c_abi_consumer/src/_bocpy_probe.c. The skeleton below shows the five places you must get right.

1. The impl struct

A heap-allocated C struct that lives outside any single Python object's lifetime. Carries its own atomic refcount and its atomic owner field.

typedef struct {
    atomic_int_least64_t refcount;   /* number of wrappers holding this impl */
    atomic_int_least64_t owner;      /* interpreter ID, or BOCPY_NO_OWNER in flight */
    /* ... your payload here ... */
} your_impl;

static your_impl *your_impl_new(void) {
    your_impl *impl = PyMem_RawMalloc(sizeof(*impl));
    if (impl == NULL) return NULL;
    atomic_store(&impl->refcount, 1);
    atomic_store(&impl->owner, bocpy_interpid());   /* born owned */
    return impl;
}

static void your_impl_incref(your_impl *impl) {
    atomic_fetch_add(&impl->refcount, 1);
}

static void your_impl_decref(your_impl *impl) {
    if (atomic_fetch_add(&impl->refcount, -1) == 1) {
        /* last holder, free the payload */
        PyMem_RawFree(impl);
    }
}

static bool your_impl_check_acquired(your_impl *impl, bool set_error) {
    if (bocpy_interpid() != atomic_load(&impl->owner)) {
        if (set_error)
            PyErr_SetString(PyExc_RuntimeError,
                            "the current interpreter does not own this type");
        return false;
    }
    return true;
}

Refcounting and ownership are independent. Any interpreter holding a wrapper drops its ref on dealloc, regardless of who currently owns the impl. The proto-Region check guards data accessors, not the lifetime of the impl itself.

2. The Python wrapper, as a heap type with per-module state

Use PyType_FromModuleAndSpec (not the static-type pattern). Store the type on per-module state and cache it in a thread_local LOCAL_STATE so callbacks can find it without walking module state.

typedef struct {
    PyTypeObject *your_type;
} your_module_state;

static thread_local your_module_state *LOCAL_STATE;

typedef struct {
    PyObject_HEAD
    your_impl *impl;
} YourObject;

Data accessors must call your_impl_check_acquired before reading the payload. Identity-only accessors (e.g. an address getter that returns (uintptr_t)impl) may skip the check — printing the address of a Region handle without being inside the Region is allowed.

3. The producer callback — declare with XIDATA_GETDATA_FUNC

Runs on the interpreter that currently owns the impl, every time something asks XIData to package one of your objects. CAS the owner field from this interpreter to BOCPY_NO_OWNER, then call XIDATA_INIT. Failing the CAS surfaces as a RuntimeError and aborts the handoff.

XIDATA_GETDATA_FUNC(_your_shared) {
    YourObject *self = (YourObject *)obj;
    your_impl *impl = self->impl;
    if (impl == NULL) {
        PyErr_SetString(PyExc_RuntimeError, "type not initialised");
        return -1;
    }

    int_least64_t expected = bocpy_interpid();
    int_least64_t desired  = BOCPY_NO_OWNER;
    if (!atomic_compare_exchange_strong(&impl->owner, &expected, desired)) {
        PyErr_Format(PyExc_RuntimeError,
                     "cannot share: owned by interpreter %lld",
                     (long long)expected);
        return -1;
    }

    XIDATA_INIT(xidata, tstate->interp, impl, obj, _new_your_object);
    return 0;
}

Why XIDATA_GETDATA_FUNC and not a hand-written signature? On older CPython the callback is (obj, xidata) only — no tstate. The macro emits a small trampoline so the body is portable across every supported CPython.

4. The consumer callback — new_object reconstruction

Runs on the interpreter that is taking ownership. CAS the owner from BOCPY_NO_OWNER to bocpy_interpid(), allocate a fresh wrapper from the local heap type, and bump the impl refcount.

static PyObject *_new_your_object(XIDATA_T *xidata) {
    your_impl *impl = (your_impl *)xidata->data;

    int_least64_t expected = BOCPY_NO_OWNER;
    int_least64_t desired  = bocpy_interpid();
    if (!atomic_compare_exchange_strong(&impl->owner, &expected, desired)) {
        PyErr_Format(PyExc_RuntimeError,
                     "cannot acquire (expected BOCPY_NO_OWNER, observed %lld)",
                     (long long)expected);
        return NULL;
    }

    PyTypeObject *type = LOCAL_STATE->your_type;
    YourObject *self = (YourObject *)type->tp_alloc(type, 0);
    if (self == NULL) {
        /* CRITICAL: roll the owner back so a retry can succeed and
         * the impl is not stranded with us as owner without a wrapper. */
        atomic_store(&impl->owner, BOCPY_NO_OWNER);
        return NULL;
    }
    self->impl = impl;
    your_impl_incref(impl);
    return (PyObject *)self;
}

The wrapper-allocation rollback is easy to forget and impossible to recover from at runtime. Write it the same time you write the success path.