Magpie Kernel Evaluator

Magpie

Use Magpie for three connected jobs:

1. Benchmark an inference workload and collect throughput, latency, and traces.
2. Analyze or compare GPU kernels for correctness and performance.
3. Drive an optimization loop from a benchmark bottleneck to source, candidate kernels, and end-to-end validation.

Describe only capabilities supported by the checked-out Magpie version. Do not infer support for an unverified ROCm, GPU, framework, or experimental integration.

Choose the workflow

Use a YAML config for reproducible or multi-step work. Use inline CLI arguments for small exploratory runs.

Preflight

Installing this skill does not install the Magpie application (magpie-eval). For a preparation-only request, read the supplied fixtures and reference.md, write the requested plan or configuration, and identify commands that remain unverified. Do not install packages or execute a workload when the user forbids it; a missing Magpie installation does not prevent preparing a plan.

Before executing a Magpie workload:

1. Check magpie --help in the intended Python environment (Python 3.10+). If the command is missing, try python -m Magpie --help in that same environment; a working module entry point can be used instead of the CLI. No module named Magpie means the application is not installed in that environment. For errors inside an installed Magpie package, diagnose the reported import or dependency failure instead.

2. If Magpie is not installed, install it in the intended environment before continuing:

   python -m pip install git+https://github.com/AMD-AGI/Magpie.git

   For an existing Magpie source checkout, use python -m pip install -e /path/to/Magpie instead. The installed skill folder and a testcase workspace are not Magpie source checkouts. Re-run magpie --help or python -m Magpie --help to verify the installation; if it fails, report the error before attempting a workload.

3. Inspect the installed interface for the selected workflow (substitute python -m Magpie if using the module entry point):

   magpie --help
   magpie analyze --help
   magpie compare --help
   magpie benchmark --help
   magpie --gpu-info

4. Check required tools, model access, GPU visibility, writable output space, and container or Ray access as applicable. Installing the Python package does not install ROCm/CUDA, profilers, or model weights.

5. Read the repository compatibility matrix before making version claims. Treat ROCm or hardware not listed there as unverified until tested.

6. Record the exact config, model revision, image, environment variables, GPU allocation, and Magpie commit for benchmark comparisons.

Analyze a kernel

Prefer a config when correctness or profiler settings matter:

magpie analyze --kernel-config path/to/kernel.yaml

For a quick single-kernel run:

magpie analyze path/to/kernel.hip --type hip --testcase "./run_test.sh"

Supported public kernel types are hip, cuda, pytorch, and triton. Use --no-perf only when the user wants correctness or execution validation without profiling.

Do not equate successful execution with numerical correctness. Supply a representative testcase whenever an optimized result will be accepted or rejected.

Compare kernel variants

Compare at least two implementations and identify the baseline explicitly:

magpie compare --kernel-config path/to/compare.yaml

Keep inputs, tolerances, warmup, iteration count, GPU allocation, and profiler settings identical across candidates. Reject candidates that fail correctness before considering performance rankings.

For PyTorch without a testcase, Magpie's built-in check only verifies that each result is finite; it does not prove numerical equivalence between variants. Require a testcase for numerical validation.

Benchmark inference

Prefer a checked-in benchmark config:

magpie benchmark --benchmark-config path/to/benchmark.yaml

The stable public CLI supports vllm, sglang, and atom. It supports direct docker and local run modes; use YAML configuration and the repository's Ray examples for distributed execution. Do not advertise integrations that exist only in internal enums or partial code paths as stable.

Enable profiling deliberately: profiler runs perturb latency and should not replace a clean baseline. Compare throughput, completed requests, TTFT, TPOT, ITL, and end-to-end latency using equivalent workloads.