Build Afm Nightly Publish

Build & Publish AFM Nightly

Build afm from scratch (works from a fresh clone), let the user test it, then publish a GitHub pre-release and update the Homebrew tap.

Usage

• /build-afm-nightly-publish — full pipeline: build + test + publish
• /build-afm-nightly-publish --skip-build — skip build, use existing release binary

Prerequisites

The publish script (Scripts/publish-next.sh) requires:

• gh CLI authenticated with push access to scouzi1966/maclocal-api
• homebrew-afm repo at ../homebrew-afm (relative to repo root) or TAP_DIR env var — auto-cloned if missing
• vesta-mac repo at ../vesta-mac (relative to repo root) or VESTA_DIR env var — required for PEP 503 wheel index update. Auto-cloned if missing.
• All build prerequisites from /build-afm (Xcode, Swift, Node.js, etc.)
• promptfoo CLI (npm install -g promptfoo or npx promptfoo) — required for the promptfoo agentic eval suite
• wrangler CLI (npm install -g wrangler) — required for Cloudflare Pages deploy of wheel index

Instructions

Step 1: Validate Environment

Run these checks and present results to the user:

# Build prerequisites
uname -m                    # must be arm64
sw_vers -productVersion     # must be 26.0+
xcode-select -p             # must point to Xcode.app
swift --version             # Swift 5.9+
git --version
node --version              # Node 18+
npm --version

# Promptfoo (required for agentic eval suite)
command -v promptfoo && promptfoo --version 2>/dev/null | head -1
# Must be installed (npm install -g promptfoo)

# Publish prerequisites
gh auth status              # must be authenticated

# CRITICAL: Verify the user is the repo owner (scouzi1966)
# This prevents non-owners from accidentally overwriting releases or the brew tap.
GH_USER=$(gh api user -q .login)
echo "GitHub user: $GH_USER"
# Must be "scouzi1966"

# Verify push (write) access to both repos
gh api repos/scouzi1966/maclocal-api -q '.permissions.push'    # must be true
gh api repos/scouzi1966/homebrew-afm -q '.permissions.push'    # must be true

# Tap repo — auto-clone if missing
TAP_DIR="${TAP_DIR:-$(cd "$(git rev-parse --show-toplevel)/.." && pwd)/homebrew-afm}"
if [ ! -f "$TAP_DIR/afm-next.rb" ]; then
  echo "Tap repo missing at $TAP_DIR — cloning..."
  gh repo clone scouzi1966/homebrew-afm "$TAP_DIR"
fi
test -f "$TAP_DIR/afm-next.rb" && echo "Tap OK: $TAP_DIR" || echo "FAILED to clone tap repo"

# vesta-mac repo — required for wheel index update. Auto-clone if missing.
VESTA_DIR="${VESTA_DIR:-$(cd "$(git rev-parse --show-toplevel)/.." && pwd)/vesta-mac}"
if [ ! -d "$VESTA_DIR/.git" ]; then
  echo "vesta-mac repo missing at $VESTA_DIR — cloning..."
  gh repo clone scouzi1966/vesta-mac "$VESTA_DIR"
fi
test -d "$VESTA_DIR/.git" && echo "vesta-mac OK: $VESTA_DIR" || echo "FAILED to clone vesta-mac"

Present as a checklist. If the GitHub user is not scouzi1966 or push access is false for either repo, STOP immediately and tell the user:

This skill publishes releases and updates the Homebrew tap for scouzi1966/maclocal-api.
Only the repository owner (scouzi1966) can run it. You are authenticated as: <username>

Do NOT proceed unless: (1) all build checks pass, (2) GitHub user has push access to BOTH repos, (3) tap repo is available.

Step 2: Build from Scratch (True Clean Build)

A nightly release must be built from a completely clean state. swift package clean is NOT sufficient — it leaves behind cached modules, package resolution state, and precompiled headers that can mask stale code:

Before running the build script, nuke all cached state:

# 1. Remove entire SPM build directory (modules, cache, resolution state — everything)
rm -rf .build

# 2. Remove Xcode DerivedData for this project (if anyone opened it in Xcode)
rm -rf ~/Library/Developer/Xcode/DerivedData/*maclocal* \
       ~/Library/Developer/Xcode/DerivedData/*MacLocal* \
       ~/Library/Developer/Xcode/DerivedData/*afm* 2>/dev/null || true

# 3. Verify clean state
test -d .build && echo "FAIL: .build still exists" || echo "OK: .build removed"

Then run the full build:

./Scripts/build-from-scratch.sh

IMPORTANT: Never add --skip-submodules, --skip-patches, or --skip-webui. This is a release build — everything must be from scratch.

Why this matters: Stale ModuleCache can cause the compiler to use old .swiftmodule files from a previous build, meaning your patches compile but the binary links against the cached (unpatched) version. Stale workspace-state.json can resolve a different version of MLX Swift than what the pin specifies. Both failures are silent — the build succeeds, the binary runs, but behavior is wrong.

If the user passed --skip-build, skip the clean and build steps, but still run all Step 2b verification checks against the existing binary:

test -x .build/arm64-apple-macosx/release/afm || test -x .build/release/afm

Step 2b: Post-Build Verification ("What Could Go Wrong")

The build script reports success, but do not trust its output alone. Independently verify every critical artifact. The build script could succeed (exit 0) while:

• Patches silently failed to apply (vendor reverted by git submodule update)
• xgrammar compiled but wasn't linked (missing from Package.swift targets)
• MLX Swift resolved a wrong version (pin not applied to Package.swift)
• Metallib bundle missing (Metal shaders won't load at runtime → crash)
• WebUI assets missing (llama.cpp web interface won't serve)
• BuildInfo.swift not restored (leaves dirty working tree)

Run all of these checks. Present results as a table. If ANY check fails, STOP and investigate before proceeding.

Check 1: Patches byte-identical to vendor targets

The patch script says "Applied" but git submodule update can silently revert files. Verify every patch file is byte-for-byte identical to its vendor target using the actual arrays from Scripts/apply-mlx-patches.sh:

python3 -c "
import os
# These arrays MUST match Scripts/apply-mlx-patches.sh — if they drift, the check is wrong.
# Read them from the script itself to stay in sync.
patches = [
  ('Qwen3VL.swift','Libraries/MLXVLM/Models/Qwen3VL.swift'),
  ('Qwen3Next.swift','Libraries/MLXLLM/Models/Qwen3Next.swift'),
  # ... all 20 entries from PATCH_FILES/TARGET_PATHS arrays ...
]
ok = fail = 0
for pf, tp in patches:
    src, tgt = f'Scripts/patches/{pf}', f'vendor/mlx-swift-lm/{tp}'
    if not os.path.exists(tgt):
        print(f'MISSING:   {pf} -> {tp}'); fail += 1
    else:
        with open(src,'rb') as a, open(tgt,'rb') as b:
            if a.read() == b.read():
                print(f'MATCH:     {pf}'); ok += 1
            else:
                print(f'MISMATCH:  {pf}'); fail += 1
print(f'\n{ok}/{ok+fail} patches verified')
"

Why this matters: If even one patch is stale, the compiled binary has upstream code instead of our optimized/fixed version. This has happened when git submodule update --init --recursive runs AFTER apply-mlx-patches.sh — it silently reverts patches.

Check 2: MLX Swift pinned AND resolved to exact version

# Check the pin in source
grep 'mlx-swift.*exact' vendor/mlx-swift-lm/Package.swift
# Must show: exact: "0.30.3"
# 0.30.4+ has SDPA NaN regression — if this shows any other version, STOP.

# Check what SPM actually resolved (the pin could say 0.30.3 but resolution used a cached different version)
python3 -c "
import json
d = json.load(open('Package.resolved'))
for p in d.get('pins', []):
    if 'mlx' in p.get('identity','').lower():
        print(f'{p[\"identity\"]}: {p[\"state\"].get(\"version\",\"?\")}')
"
# Must show: mlx-swift: 0.30.3
# If version differs from pin, the resolution is stale — this is exactly what rm -rf .build prevents.

Why this matters: The pin in Package.swift is a request, but Package.resolved is what was actually fetched and compiled against. A stale workspace-state.json or Package.resolved from a previous build can cause SPM to use a cached resolution even after the pin changes. Nuking .build/ in Step 2 prevents this, but verify anyway.

Check 3: xgrammar submodule present and at expected version

git submodule status vendor/xgrammar
# Must show a commit hash, NOT a '-' prefix (which means uninitialized)
cd vendor/xgrammar && git describe --tags --always && cd -
# Must show v0.1.32 or the expected pinned tag

Why this matters: xgrammar is a C++ library compiled from source. If the submodule is missing or at the wrong version, the EBNF grammar constraint feature either doesn't exist or has different behavior.

Check 4: xgrammar symbols linked into the binary

# Verify xgrammar C++ was compiled and linked (not just present as source)
strings .build/arm64-apple-macosx/release/afm | grep -c 'xgrammar/cpp/'
# Must be > 0 (typically 10+)

# Verify our Swift XGrammarService wrapper is in the binary
strings .build/arm64-apple-macosx/release/afm | grep 'XGrammarService'
# Must show: XGrammarService, _TtC11MacLocalAPI15XGrammarService, etc.

# Verify xgrammar C++ symbols are actually linked
nm -a .build/arm64-apple-macosx/release/afm 2>/dev/null | grep -c 'xgrammar'
# Must be > 0 (typically 30+)

Why this matters: xgrammar could be in the source tree but excluded from the Swift Package Manager target graph. The binary would build fine but grammar-constrained decoding would silently fail at runtime.

Check 5: Metallib bundle present

METALLIB=".build/arm64-apple-macosx/release/AFMKit_AFMKitMLX.bundle/default.metallib"
test -f "$METALLIB" && echo "OK: metallib $(du -h "$METALLIB" | cut -f1)" || echo "FAIL: metallib missing"
# Must exist and be > 1MB (typically ~3.7MB)

Why this matters: Without the metallib, MLX GPU kernels can't load. The server starts but crashes on first inference. The build script checks this, but verify independently.

Check 6: WebUI assets present

test -f "Resources/webui/index.html.gz" && echo "OK: webui assets" || echo "FAIL: webui missing"

Why this matters: The llama.cpp web UI is served at / — without it, browser access shows nothing.

Check 7: BuildInfo.swift is clean (not left with injected SHA)

grep 'static let version' Sources/AFMKit/BuildInfo.swift
# Must show the base version like: static let version: String? = "v0.9.7"
# Must NOT show a commit SHA like: static let version: String? = "v0.9.7-3d71b40"
git diff Sources/AFMKit/BuildInfo.swift
# Must show no diff (file restored to committed state)

Why this matters: The build script injects the git SHA into BuildInfo.swift during compilation then restores it. If restore fails, the working tree is dirty and the next git commit could accidentally commit the injected version.

Check 8: Binary is stripped and reasonable size

ls -lh .build/arm64-apple-macosx/release/afm
# Size should be 30-50MB for a stripped release binary
# If > 100MB, it's likely unstripped (debug symbols included)
nm -gU .build/arm64-apple-macosx/release/afm 2>/dev/null | wc -l
# Stripped binary has minimal external symbols (< 500 typically)
# Unstripped has thousands

Check 9: Relocated binary does NOT crash (pip install simulation)

This is the most critical distribution check. SPM auto-generates resource_bundle_accessor.swift with a hardcoded absolute build path. If any code path calls Bundle.module, the binary will fatalError when installed via pip or Homebrew (because the build path no longer exists). This has shipped broken nightlies before.

# Simulate pip install: copy binary + loose metallib to a temp dir (NO SPM bundle directory)
TMPDIR=$(mktemp -d)
cp .build/arm64-apple-macosx/release/afm "$TMPDIR/"
cp .build/arm64-apple-macosx/release/AFMKit_AFMKitMLX.bundle/default.metallib "$TMPDIR/"

# Must NOT crash with "could not load resource bundle" fatalError
MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
  "$TMPDIR/afm" mlx -m mlx-community/Qwen3.5-35B-A3B-4bit -s "hello" --max-tokens 5 2>&1 | head -3
EXIT_CODE=${PIPESTATUS[0]}
rm -rf "$TMPDIR"

if [ "$EXIT_CODE" -ne 0 ]; then
  echo "FAIL: Relocated binary crashed (exit $EXIT_CODE)"
  echo "This means Bundle.module fatalError is still reachable."
  echo "Check MLXMetalLibrary.swift — it must NOT call Bundle.module."
else
  echo "PASS: Relocated binary works"
fi

Why this matters: This is the exact layout pip creates: macafm_next/bin/afm + macafm_next/bin/default.metallib. If this test fails, every pip user gets a crash on first run. This check is non-negotiable — if it fails, do NOT publish.

Root cause if it fails: Someone added a Bundle.module call somewhere in the codebase. Search for it:

grep -r 'Bundle\.module' Sources/ --include='*.swift'
# Must return ZERO results (only comments allowed)

Check 10: No Bundle.module calls in source code

# Bundle.module uses SPM's auto-generated accessor which fatalError's on relocated binaries.
# It must NEVER be called from our code. Comments referencing it are OK.
HITS=$(grep -r 'Bundle\.module' Sources/ --include='*.swift' | grep -v '//' | grep -v '^\s*//' | wc -l)
if [ "$HITS" -gt 0 ]; then
  echo "FAIL: Found $HITS Bundle.module call(s) in source code"
  grep -rn 'Bundle\.module' Sources/ --include='*.swift' | grep -v '//'
  echo "This WILL crash when installed via pip or Homebrew."
else
  echo "PASS: No Bundle.module calls"
fi

Why this matters: Even a single Bundle.module call anywhere in the code path triggers the auto-generated fatalError. This is a regression guard — any future code that adds Bundle.module will be caught here before it ships.

Check 11: Info.plist embedded with privacy usage descriptions

macOS 26 SIGABRTs any process that requests privacy-sensitive APIs (Speech Recognition, microphone, camera, contacts, etc.) without a matching *UsageDescription key in its Info.plist. Currently required for PR #107's Apple Speech feature (afm speech, POST /v1/audio/transcriptions, chat input_audio content parts). Any future privacy-API integration needs its key added here too.

BIN=.build/arm64-apple-macosx/release/afm

# Verify __TEXT,__info_plist section exists (embedded via Package.swift linker flags)
if otool -l "$BIN" | grep -q '__info_plist'; then
  PLIST_SIZE=$(otool -l "$BIN" | grep -A4 __info_plist | grep 'size' | awk '{print $2}')
  echo "PASS: __info_plist section present ($PLIST_SIZE bytes)"
else
  echo "FAIL: Missing __TEXT,__info_plist section"
  echo "Check Package.swift linker flags (-Xlinker -sectcreate ...) and Sources/AFMCLI/Info.plist"
fi

# Verify NSSpeechRecognitionUsageDescription key is present
if strings "$BIN" | grep -q 'NSSpeechRecognitionUsageDescription'; then
  echo "PASS: NSSpeechRecognitionUsageDescription key embedded"
else
  echo "FAIL: NSSpeechRecognitionUsageDescription missing from embedded plist"
  echo "afm speech / /v1/audio/transcriptions will SIGABRT on macOS 26"
fi

# Verify plist structure is parseable (not corrupted during build)
plutil -lint Sources/AFMCLI/Info.plist

Why this matters: Without the embedded plist, running afm speech -f foo.wav (or any endpoint that calls SFSpeechRecognizer) crashes before returning any output. The build script Scripts/build-from-scratch.sh already enforces this — but also check here so the publish flow fails loudly if someone bypasses the build script or if Package.swift's linker flags get reverted in a merge.

Root cause if it fails:

1. Sources/AFMCLI/Info.plist was deleted or renamed
2. Package.swift's linkerSettings lost the -Xlinker -sectcreate -Xlinker __TEXT -Xlinker __info_plist -Xlinker … flags
3. Someone added a new privacy-API usage (microphone, camera) without adding the corresponding *UsageDescription key to Info.plist

Full plist must also include CFBundleIdentifier, CFBundleName, CFBundleExecutable — these establish TCC identity. Changing CFBundleIdentifier later would force existing users to re-grant Speech Recognition permission.

Check 12: Report all vendor/submodule pin levels

Present the exact version of every submodule and SPM dependency so the user can verify the build reproduces the expected dependency tree. Also fetch the latest release tag from each upstream repo to show if we're behind.

echo "=== Git Submodules ==="
git submodule status

echo "=== SPM Resolved Versions ==="
python3 -c "
import json
d = json.load(open('Package.resolved'))
for p in sorted(d.get('pins', []), key=lambda x: x.get('identity','')):
    v = p['state'].get('version') or p['state'].get('revision','?')[:12]
    print(f'  {p[\"identity\"]}: {v}')
"

echo "=== Upstream Latest Releases ==="
for repo in ml-explore/mlx-swift ml-explore/mlx-swift-lm mlc-ai/xgrammar ggml-org/llama.cpp huggingface/swift-transformers huggingface/swift-huggingface; do
  tag=$(gh api "repos/$repo/releases/latest" -q '.tag_name' 2>/dev/null || echo "?")
  echo "  $repo: $tag"
done

This is informational — no pass/fail. But if a resolved version is unexpected (e.g., mlx-swift != 0.30.3), STOP.

Present verification results

Then present two separate tables for vendor pins:

Git Submodules:

SPM Dependencies (from Package.resolved):

Populate the "Upstream Latest" column by querying gh api repos/OWNER/REPO/releases/latest -q '.tag_name'. This lets the user see at a glance if we're behind upstream on any dependency.

If ANY check fails, STOP. Do not proceed to user testing or publishing.

Step 3: Present Binary and Enter Test/Fix/Rebuild Loop

After all verification checks pass, get the binary path and version:

BIN=".build/arm64-apple-macosx/release/afm"
[ -x "$BIN" ] || BIN=".build/release/afm"
echo "Binary: $(cd "$(dirname "$BIN")" && pwd)/$(basename "$BIN")"
$BIN --version

Report to the user:

• Binary path (absolute)
• Version string
• Verification results table (from Step 2b)

Then use AskUserQuestion to pause and let the user decide what to do next:

Question: "The build is verified. What would you like to do?"

Options:

1. "Publish as-is" — Skip testing, go straight to GitHub release and tap update
2. "Run tests" — Run automated tests, then decide (see test scope question below)
3. "I'll test manually" — Pause here while the user tests the binary themselves
4. "Cancel" — Abort without publishing

If user selects "Run tests"

First, list available models in the cache and let the user pick:

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache ./Scripts/list-models.sh

Question: "Which model to test with?"

Present the available models as options (show model name and size). The user picks one.

Then ask the test scope:

Question: "Which tests to run?"

Options:

1. "Assertions only (all tiers including unit)" — Run /test-afm-assertions with full tier (deterministic pass/fail tests, ~15 min/model)
2. "Comprehensive only" — Run /test-macafm smart analysis (AI-scored quality evaluation)
3. "Both" — Run assertions first, then comprehensive (most thorough, ~30 min/model)
4. "Full nightly suite" — Run assertions + comprehensive + promptfoo agentic evals (most complete, ~60 min)

Invoke the appropriate skill(s) with the selected model. Do NOT re-ask the model question — pass it through to the test skill(s).