LLM inference and serving

Reference desk for NVIDIA Nemotron 3 Super — architecture, training data, recipes (pretrain/SFT/RL/eval/quantization), and deployment notes.

NVIDIA-NeMo/Nemotron

Add support for a new Vision-Language Model (VLM) to AutoRound, including multimodal block handler, calibration dataset template, and special model handling.

intel/auto-round

GGUF format and llama.cpp quantization for efficient CPU/GPU inference.

Orchestra-Research/AI-Research-SKILLs

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.

Orchestra-Research/AI-Research-SKILLs

A template skill demonstrating the PAI SKILL.md format. An agent skill from nirholas/PAI.

nirholas/PAI

QAI ModelBuilder. An agent skill from qualcomm/qai-appbuilder.

qualcomm/qai-appbuilder

Split a multi-call LM workflow by cognitive load, not by accuracy: let one strong model make the few reasoning decisions and a cheap model do the many mechanical executions (Aider architect+editor…

agentsope/SkillAlchemy

Databricks Model Serving endpoint lifecycle and ops. An agent skill from databricks/databricks-agent-skills.

databricks/databricks-agent-skills

Find evidence-backed simplification candidates in vLLM-Omni and, when requested, turn them into focused proposals or code changes.

vllm-project/vllm-omni

Run RecIF beam×concurrency eval and FlashRec vs SGLang/vLLM/TRT-LLM baselines.

sohu-mptc/FlashRec

Benchmark vLLM or OpenAI-compatible serving endpoints using vllm bench serve.

vllm-project/vllm-skills

Explains where HOT-Step generation time goes (LM/DiT/VAE), how the TensorRT paths activate, how to benchmark from logs, and which knobs trade quality for speed.

scragnog/HOT-Step-CPP

Run, resume, monitor, diagnose, and report Quark Quant-Perf workflows for PyTorch and HuggingFace transformers models.

amd/Quark

Register, list, get, and manage LLM models in OCI AI Quick Actions (AQUA) using the ADS SDK.

oracle/accelerated-data-science

Guide a tester through hipfire bring-up, serve smoke, claim-scoped harnesses, and benchmark reporting on AMD RDNA/CDNA GPUs.

warpfront/hipfire

Bump Heron version via the VERSION-file SSOT. An agent skill from Netis/heron.

Netis/heron

A skill your agent uses when adding, debugging, or validating a bring-your-own VLM in VSS RT-VLM, including custom Hugging Face or NGC checkpoints, vLLM adapters or plugins, model shims, and…

NVIDIA-AI-Blueprints/video-search-and-summarization

Review or prepare a pull request for the AutoRound repository — checks registration points for new data types/backends/VLMs, validates Chinese translation parity for modified markdown files…

intel/auto-round

Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.

Orchestra-Research/AI-Research-SKILLs

Post-training 4-bit quantization for LLMs with minimal accuracy loss.

Orchestra-Research/AI-Research-SKILLs

Half-Quadratic Quantization for LLMs without calibration data.

Orchestra-Research/AI-Research-SKILLs

Provides guidance for enterprise-grade RL training using miles, a production-ready fork of slime.

Orchestra-Research/AI-Research-SKILLs

Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace.

Orchestra-Research/AI-Research-SKILLs

High-performance RLHF framework with Ray+vLLM acceleration. An agent skill from Orchestra-Research/AI-Research-SKILLs.

Orchestra-Research/AI-Research-SKILLs

Loads large language models in 8-bit or 4-bit with bitsandbytes so they fit smaller GPUs, and sets up QLoRA fine-tuning on a 4-bit base model.

Orchestra-Research/AI-Research-SKILLs

Covers serving LLMs with SGLang, whose RadixAttention reuses cached prefixes, and constraining output to JSON, regex or grammar for agent and tool-calling workloads.

Orchestra-Research/AI-Research-SKILLs

Explains three ways to speed up LLM inference: draft-model speculative decoding, Medusa heads and lookahead decoding with Jacobi iteration, and when each one fits.

Orchestra-Research/AI-Research-SKILLs

Review and adapt MetaX attention backends, MLA, sparse indexers, cache layouts, and their kernel wrappers against a target vLLM revision and the actually installed MetaX component APIs.

MetaX-MACA/vLLM-metax

Summarize a document at a requested length. An agent skill from nirholas/PAI.

nirholas/PAI

This is an experimental Skill. An agent skill from GoogleCloudPlatform/accelerated-platforms.

GoogleCloudPlatform/accelerated-platforms

Review llama.cpp changes against project conventions and common reviewer pitfalls before a PR.

JakeATX/llamAmpere

Run and troubleshoot the inference backends on the DGX Spark pair that FastLLM proxies to — starting or stopping models with vLLM, SGLang or sparkrun, choosing memory and speculative-decoding…

azrtydxb/Fastllm-proxy

Productionize a vLLM-Omni diffusion model after its Day-0 vertical slice works.

vllm-project/vllm-omni

Serves an LLM on a supported AMD EPYC server CPU using vLLM with zentorch, in Docker, Podman, or conda.

amd/skills

Analyze and optimize vllm-rlt inference performance using reproducible unprofiled benchmarks, paired ops-only/full profiles, source-level attribution, and correctness checks.

ThinkFlowLab/vllm-rlt

Deploy vLLM to Kubernetes (K8s) with GPU support, health probes, and OpenAI-compatible API endpoint.

vllm-project/vllm-skills

Writes a Triton kernel pattern for Ascend NPU that gathers several indexed token rows per program into an on-chip buffer and stores one contiguous output tile, for MoE-style token reordering.

Krusty84/triton-ascend-agent-dev-kit

Routes a NanoClaw agent group to a local Ollama model instead of the Anthropic API, using environment overrides and an optional block on Anthropic hosts.

nanocoai/nanoclaw

Sync documentation with project state using ICAV workflow. An agent skill from Netis/heron.

Netis/heron

Build and query a local vector database + GraphRAG over your entire reference library (Zotero or a PDF folder) for literature review.

joshzyj/open-scholar-skill

Breaks LLM torch profiler traces down by forward pass, layer and kernel, with timing tables and Perfetto time ranges for the layers you want to inspect.

BBuf/AI-Infra-Auto-Driven-SKILLS

Audit and adapt monkey patches in vllmmetax/patch/ against a target upstream revision.

MetaX-MACA/vLLM-metax

Explains RWKV, a hybrid that trains in parallel like a GPT and runs inference like an RNN with constant memory per token, plus usage, fine-tuning and troubleshooting.

Orchestra-Research/AI-Research-SKILLs

Generate and run tests for vllm-project/vllm-omni with CI-aligned levels and markers; wire new tests into Buildkite (test-ready.yml for L1/L2, test-merge.yml for L3, test-nightly.yml for L4).

vllm-project/vllm-omni

Entry point for hosting a model on Amazon SageMaker: asks a few questions, picks a deployment pathway and hands off to the specialist skills.

huggingface/skills

Author a new ShapeShifter graph-transformation pass for AMD Quark (ONNX or PyTorch) so it conforms to the pass framework's conventions and auto-registers.

amd/Quark

Quick install and deploy vLLM, start serving with a simple LLM, and test OpenAI API.

vllm-project/vllm-skills

Autonomously optimizes end-to-end LLM inference throughput on AMD Instinct GPUs and reports a validated gain, using the Hyperloom multi-agent optimizer.

amd/skills