Test Macafm

test-macafm

Run the maclocal-api test suite: automated pass/fail assertions and smart analysis (the smart suite's AI judge is opt-in — default off; ask the user before enabling it).

Triggers

Use this skill when the user asks to:

• Test or validate the server (e.g., "run the tests", "test AFM", "validate the build")
• Regression check after code changes
• Onboard a new model (verify it works correctly with the server)
• Release check before tagging or pushing
• Benchmark or profile model performance

First Questions to Ask

1. Model — Which model to test? (Ask if not specified. Default: whatever's loaded.)
2. Tier — smoke / standard / full? (Suggest based on context.)
3. Binary path — Default .build/release/afm. Ask if user has a custom build location.
4. Port — Default 9998. Ask if user's server is on a different port.
5. Server running? — Is the server already running, or should tests start it?

Tier Decision Tree

Quick guide:

• "Just run a quick test" → smoke
• "Test before merging" → standard
• "Full release validation" or "onboard new model" → full
• User doesn't specify → suggest standard

Execution Workflow

1. Build Check

# Ensure release build is current
swift build -c release

2. Start Server (if not running)

MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
  .build/release/afm mlx -m MODEL --port 9998 \
  --tool-call-parser afm_adaptive_xml \
  --enable-prefix-caching \
  --enable-grammar-constraints &
# Wait for server to be ready
until curl -sf http://127.0.0.1:9998/v1/models >/dev/null 2>&1; do sleep 1; done

Recommended flags for testing:

• --tool-call-parser afm_adaptive_xml — best tool call parser with JSON-in-XML fallback
• --enable-prefix-caching — 67-79% prompt token savings on repeated requests
• --enable-grammar-constraints — EBNF constrained decoding forces valid XML tool calls, improving success from 60% to 100% on realistic workloads

3. Run Automated Assertions

./Scripts/test-assertions.sh --tier TIER --model MODEL --port 9998

Interpret results immediately. If any FAIL, investigate before proceeding.

4. Run Smart Analysis (full tier only)

The smart analysis harness manages its own server (port 9877) — do NOT pass --port. It uses test-llm-comprehensive.txt which has an [all] baseline prompt and [@ label] template sections.

The AI judge is OPT-IN — default OFF. By default, run the smart suite WITHOUT an AI judge: it executes every prompt and records the model's raw outputs to the report for manual review, with no claude/codex scoring. Before running, ask the user (e.g. via AskUserQuestion) whether to enable the AI judge — it adds latency/cost and invokes an external CLI:

"Run the smart suite with an AI judge (claude) scoring each response, or without it (just record outputs for manual review)? Default: without."

Default — no AI judge (records outputs only; omit --smart entirely):

AFM_BIN=.build/release/afm ./Scripts/mlx-model-test.sh \
  --model MODEL \
  --prompts Scripts/test-llm-comprehensive.txt

Only if the user opts in — append --smart 1:claude (the --smart flag accepts a batch mode prefix and tool list):

AFM_BIN=.build/release/afm ./Scripts/mlx-model-test.sh \
  --model MODEL \
  --prompts Scripts/test-llm-comprehensive.txt \
  --smart 1:claude

Smart analysis options (only when the AI judge is enabled):

• --smart claude or --smart codex — batch mode 0 (one big swoop, may fail on large test suites)
• --smart 1:claude or --smart 1:codex — batch mode 1 (test-by-test, more reliable)
• --smart 1:claude,codex — run multiple AI judges
• --tests 1,5,10 — run only specific test numbers (1-indexed)

Note: The [all] prompt runs for every test variant. With high max_tokens (e.g., 32768 on code tests), thinking models may generate very long reasoning for the baseline prompt. Total run time for full suite: ~45-90 min depending on model speed.

5. Run GPU Shader Profile (full tier, or when investigating perf)

Generates an interactive HTML report with measured DRAM bandwidth, GPU utilization/power timelines, and per-kernel Metal shader names from xctrace Shader Timeline.

One-time setup (creates custom Instruments template with Shader Timeline enabled):

python3 Scripts/create-shader-template.py

Run the profile (no server needed — uses single-prompt mode):

python3 Scripts/gpu-profile-report.py MODEL [max_tokens] [prompt]
# Default: 4096 tokens, built-in GPU analysis prompt
# Example: python3 Scripts/gpu-profile-report.py mlx-community/Qwen3.5-35B-A3B-4bit

This does everything automatically:

1. Warms up mactop (bandwidth monitor, no sudo)
2. Runs inference with --gpu-profile --gpu-trace 15
3. Collects 300ms bandwidth/GPU/power samples via PTY during inference
4. Extracts shader kernel names from the xctrace trace
5. Generates /tmp/afm-gpu-profile.html and opens in browser

Or use individual flags on any AFM invocation:

afm mlx -m MODEL --gpu-profile -s "prompt"           # Zero-overhead stats
afm mlx -m MODEL --gpu-profile-bw -s "prompt"        # + mactop bandwidth (~5s)
afm mlx -m MODEL --gpu-trace 10 -s "prompt"          # xctrace shader trace

Live bandwidth monitor (run in separate terminal during server requests):

./Scripts/gpu-profile.sh bandwidth

What the report shows:

• Device info (chip, memory, architecture)
• Prefill/decode tok/s with exact timing
• Memory breakdown (model weights vs KV cache)
• DRAM bandwidth timeline chart (measured via mactop)
• GPU utilization & power timeline chart
• Per-kernel Metal shader names (from Shader Timeline)
• Exact command line for reproducibility

What to look for:

• GPU utilization <100% during decode → CPU-GPU pipeline bubbles
• Bandwidth utilization >80% → memory-bound, kernel optimization won't help
• Bandwidth utilization <20% with MoE model → normal (only active experts read)
• Key kernels: affine_qmv_fast (decode bottleneck), steel_gemm_fused (prefill), sdpa_vector (attention)

5b. API-Based GPU Profiling (per-request, no CLI flags needed)

Clients can request GPU profiling data via the X-AFM-Profile HTTP header. No server flags required — works on any running AFM server.

Two levels:

# Summary: GPU power, memory, bandwidth, tok/s
curl http://127.0.0.1:9999/v1/chat/completions \
  -H "X-AFM-Profile: true" \
  -d '{"model":"m","messages":[{"role":"user","content":"Hi"}]}'

# Extended: summary + 300ms time-series samples (for charts/dashboards)
curl http://127.0.0.1:9999/v1/chat/completions \
  -H "X-AFM-Profile: extended" \
  -d '{"model":"m","messages":[{"role":"user","content":"Hi"}]}'

Response fields (afm_profile):

• gpu_power_avg_w / gpu_power_peak_w — GPU power via native IOReport (no mactop)
• memory_weights_gib / memory_kv_gib / memory_peak_gib — memory breakdown in GiB
• prefill_tok_s / decode_tok_s — throughput
• est_bandwidth_gbs — DRAM bandwidth from IOReport power (calibrated at startup via MLX GPU stress)
• chip / theoretical_bw_gbs — hardware context
• gpu_samples — number of 300ms readings taken

Extended adds (afm_profile_extended):

• summary — same as afm_profile
• samples[] — per-300ms readings: {t, bw_gbs, gpu_pct, gpu_power_w, dram_power_w}

How it works internally:

• IOReport Energy Model + GPU Stats channels sampled every 300ms via DispatchSource timer
• DRAM bandwidth derived from DRAM power using chip-specific calibration constant
• Calibration runs once at startup: 1 GiB MLX GPU stress test (~2s, async, non-blocking)
• Per-request isolation: concurrent profiled requests are guarded (second request skips gracefully)
• Zero overhead when header not sent (one string lookup per request)
• Works for both streaming (SSE event before [DONE]) and non-streaming

What to look for:

• gpu_power_peak_w ~28W during decode on M3 Ultra (matches mactop)
• est_bandwidth_gbs ~170-180 GB/s for Qwen3.5-35B-A3B-4bit (21% of 800 GB/s theoretical)
• Short requests (<300ms): at least 1 sample (timer first-fires at 100ms)
• afm_profile absent from response when header not sent (no null pollution)

6. Run Promptfoo Agentic Evals (full tier, or when validating tool calling / structured output)

The promptfoo agentic eval suite tests AFM's tool-calling and structured-output across multiple server configurations and real-world agent framework schemas. It manages its own server lifecycle.

Prerequisites: promptfoo CLI must be installed (npm install -g promptfoo).

Run the full suite:

AFM_MODEL=MODEL \
AFM_BINARY=.build/arm64-apple-macosx/release/afm \
MACAFM_MLX_MODEL_CACHE=/Volumes/edata/models/vesta-test-cache \
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh all

Run individual suites:

# Just structured output tests
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh structured

# Just tool calling (all 3 parser profiles)
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh toolcall

# Just grammar constraint validation (8 server phases)
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh grammar-constraints

# Just one agent framework
./Scripts/feature-promptfoo-agentic/run-promptfoo-agentic.sh opencode

Available modes: all, structured, structured-stress, toolcall, toolcall-quality, grammar-constraints, agentic, frameworks, opencode, pi, openclaw, hermes, default, adaptive-xml, adaptive-xml-grammar

Suite Coverage (~137 test cases across 16 configs)

Server Profiles (managed automatically by the script)

Custom Provider & Judges

• providers/afm_provider.mjs — Custom promptfoo provider with two transports: api (OpenAI-compatible HTTP) and cli-guided-json (direct binary invocation). Supports extract modes: content, tool_calls, normalized_message, full_response. Captures responseHeaders for grammar header assertions.
• judges/assert-grammar-header.mjs — Validates X-Grammar-Constraints response header: expects "downgraded" when grammar not available, absent when grammar active.
• judges/classify-failures.mjs — Post-run AI-based failure classifier: categorizes each failure as afm_bug (server/protocol), model_quality (wrong tool/args), or harness_bug (false negative).

Environment Variables

Output

JSON reports per suite+profile in $AFM_PROMPTFOO_OUT_DIR:

• structured-MODEL_SLUG.json
• toolcall-{default,adaptive-xml,adaptive-xml-grammar}-MODEL_SLUG.json
• grammar-{schema,tools}-{no-grammar,grammar-enabled,adaptive-xml,concurrent,prefix-cache}-MODEL_SLUG.json
• {agentic,frameworks,opencode,pi,openclaw,hermes}-{default,adaptive-xml,adaptive-xml-grammar}-MODEL_SLUG.json

7. Review Reports

• Assertion report: test-reports/assertions-report-*.html
• Smart analysis: test-reports/smart-analysis-{tool}-*.md
• HTML report: test-reports/mlx-model-report-*.html
• GPU profile: /tmp/afm-gpu-profile.html (+ /tmp/afm-metal.trace for Instruments)
• JSONL data: test-reports/assertions-report-*.jsonl, test-reports/mlx-model-report-*.jsonl
• Promptfoo evals: $AFM_PROMPTFOO_OUT_DIR/{suite}-{profile}-MODEL_SLUG.json (default: /Volumes/edata/promptfoo/data/maclocal-api/current/)

7. Stop Server (if we started it)

kill %1  # or whatever the background job is

Interpreting Results

Assertion Test Failures