Llama Cpp

llama.cpp

Purpose

• Use this skill to run quantized GGUF models on laptops, workstations, and edge systems.
• Prefer it when you need portable local inference without a heavyweight serving stack.
• llama.cpp is especially useful for CPU-first deployments, low-cost GPU offload, and offline workflows.
• Python users typically access it through llama-cpp-python.

Install

• Fastest path for Python users:

pip install llama-cpp-python

• Build from source when you need custom acceleration backends or tighter platform control.
• Source builds are common for CUDA, Metal, ROCm, Vulkan, and CPU-tuned environments.
• Confirm the package imports successfully:

python -c "from llama_cpp import Llama; print('ok')"

Model Format

• llama.cpp primarily uses the GGUF model format.
• GGUF packages tokenizer metadata, architecture settings, and quantized weights in one artifact.
• Choose a GGUF variant that matches your hardware budget and quality target.

Download GGUF Models

• Hugging Face is the standard source for GGUF checkpoints.

• Common repos include:

• bartowski/*

• TheBloke/*

• Typical examples:

• bartowski/Llama-3.1-8B-Instruct-GGUF

• TheBloke/Mistral-7B-Instruct-v0.2-GGUF

• bartowski/Qwen2.5-7B-Instruct-GGUF

• Store the downloaded file locally, for example:

• models/llama-3.1-8b-instruct-q4_k_m.gguf

Quantization Levels

• Q4_K_M: best balance for many local deployments

• Q5_K_M: more quality, more RAM or VRAM

• Q8_0: highest quality among common quantized options

• Q2_K: very small, but quality drops sharply

• Start with Q4_K_M unless you already know the task is quality-sensitive.

• Move to Q5_K_M or Q8_0 for coding, reasoning, or long-form generation where quality matters more.

• Use Q2_K only for extreme memory constraints or experiments.

Basic Python Usage

from llama_cpp import Llama

llm = Llama(
    model_path="models/llama-3.1-8b-instruct-q4_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=0,
)

output = llm(
    "Explain why GGUF is useful for local inference.",
    max_tokens=256,
    temperature=0.7,
)

print(output["choices"][0]["text"])

Important Init Parameters

• model_path: path to the .gguf file on disk

• n_gpu_layers: number of transformer layers to offload to GPU

• n_ctx: context size, constrained by the model and available memory

• n_threads: CPU worker threads for prompt processing and generation

• These four parameters are the first tuning knobs to adjust for almost every deployment.

Initialization Guidance

• Keep model_path on a local SSD when possible.
• Set n_threads close to the number of performant CPU cores, not necessarily total logical threads.
• Increase n_ctx only after checking memory pressure.
• Increase n_gpu_layers gradually if the model fails to load or performance is unstable.

Generation Call

• Basic call shape:

result = llm(
    "Write a short checklist for running a local LLM service.",
    max_tokens=256,
    temperature=0.7,
)

• Access the text with:

print(result["choices"][0]["text"])

• Keep max_tokens bounded for interactive usage.
• Use lower temperatures for summarization, extraction, and tool-style tasks.

Chat Format

• For instruction-tuned models, prefer the chat API:

from llama_cpp import Llama

llm = Llama(
    model_path="models/qwen2.5-7b-instruct-q4_k_m.gguf",
    n_ctx=8192,
    n_gpu_layers=20,
    n_threads=8,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are concise and technical."},
        {"role": "user", "content": "List three tradeoffs of 4-bit quantization."},
    ],
    max_tokens=256,
    temperature=0.4,
)

print(response["choices"][0]["message"]["content"])

• Use the chat API when the model card says the checkpoint is chat-tuned or instruct-tuned.
• Keep prompt templates aligned with the model family if outputs seem malformed.

Streaming Output

• Stream tokens for responsive CLI or web applications:

stream = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are concise."},
        {"role": "user", "content": "Describe GPU offload in llama.cpp."},
    ],
    max_tokens=128,
    temperature=0.3,
    stream=True,
)

for chunk in stream:
    delta = chunk["choices"][0].get("delta", {})
    text = delta.get("content")
    if text:
        print(text, end="", flush=True)

• Streaming is useful for chat UIs, terminals, and server-sent event bridges.

GPU Offload

• n_gpu_layers=-1 means full GPU offload when supported by the backend and hardware.
• You can also offload only the first N layers:

llm = Llama(
    model_path="models/mistral-7b-instruct-q5_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=-1,
)

• If full offload fails, try a partial value like 20, 30, or 40.
• Partial offload is common on consumer GPUs with limited VRAM.

CPU-Only Example

from llama_cpp import Llama

llm = Llama(
    model_path="models/phi-3-mini-q4_k_m.gguf",
    n_ctx=4096,
    n_threads=12,
    n_gpu_layers=0,
)

• CPU-only mode is viable for smaller models and latency-tolerant tasks.
• It is a good fit for offline assistants, batch summarization, and test environments.

Context Size

• n_ctx controls the context window in tokens.
• Larger values increase RAM or VRAM usage.
• The effective maximum depends on the model architecture, quantization, and rope scaling setup.
• Do not assume every GGUF file supports the same long context as the original FP16 checkpoint.