Engine Mlx Lm

mlx-lm server (Apple silicon)

Official docs, read 2026-09-29: SERVER.md · server.py (flags, routes) · README · Hugging Face cache variables · Hugging Face cache layout Tested: mlx-lm 0.31.3 (the latest on PyPI that day), uv venv with Python 3.12, MacBook Pro M1 Pro 32 GB, macOS 26.6, 2026-09-29. Tags: [doc] official source above, [run] seen on the tested machine, [?] unverified.

When to use it

• fleet models lists a model with format: mlx (an MLX-converted folder: config.json with a quantization block) on a Mac: this is its engine. Grid's own engine reads only GGUF.
• format: safetensors (a plain Hugging Face model) on a Mac: mlx-lm served one directly [run]; vLLM and SGLang are for NVIDIA/AMD servers.
• If LM Studio is installed and already holds the MLX model, LM Studio can serve it with nothing new installed (engine-lm-studio). If a GGUF of the same model exists, prefer Grid's engine.
• The docs call the server "not recommended for production" (basic security only) [doc]: bind 127.0.0.1.
• A GGUF of the same model already on disk is served by Grid's engine; do not download an MLX copy of it.
• mlx-lm installed and no MLX model yet: "$GRID_FLEET" candidates mlx (see run-local-model).
• Verified end to end [run]: started offline on a local snapshot, fleet verify passed ready, answer, tool call and speed; grid join … --at http://127.0.0.1:P/v1 -m <snapshot path> --advertise-as NAME joined in 6 s and the relay answered through the grid.

Where its models live / how to list them

• Hugging Face cache: $HF_HUB_CACHE, else $HF_HOME/hub, else ~/.cache/huggingface/hub [doc]. Layout models--<org>--<name>/snapshots/<commit>/ with files linked into blobs/ [doc].
• An MLX folder has config.json with a quantization block (bits, group_size) or sits in an mlx-community repo; a plain model has config.json and *.safetensors without it. fleet models reports both, including folders outside the cache (~/models, LM Studio's folder).
• Server side: GET /v1/models lists every cached repo as org/name plus the served path [doc][run].

Installed? Running?

• Installed: fleet models lists mlx-lm under installed engines — on PATH or in ~/.grid/envs/mlx-lm. Install there and nowhere else, never into the system Python (a slow step: ask for the go-ahead first): uv venv --python 3.12 ~/.grid/envs/mlx-lm && uv pip install --python ~/.grid/envs/mlx-lm/bin/python mlx-lm [run] (pip install mlx-lm or conda install -c conda-forge mlx-lm [doc]). ENV below is that folder.
• Running: fleet models → an engine on 8080 labelled openai-compatible (it has no owned_by) whose models are Hugging Face ids [run].

Start an already-downloaded model (never downloads)

Port P from outside machine.listeningPorts (8090 was taken by another app during testing [run]):

"$GRID_FLEET" serve mlx-P --env HF_HUB_OFFLINE=1 -- ENV/bin/mlx_lm.server \
  --model <snapshot dir or cached org/name> --host 127.0.0.1 --port P --max-tokens 32768 \
  --chat-template-args '{"enable_thinking":false}'

• fleet serve starts it in its own session, so it outlives your shell; log in run/mlx-P.log, PID in run/mlx-P.pid. Under the agent's shell a plain nohup … & died with an empty log when the command returned, while a fleet serve process was still alive in a later command [run]. Never launchd.

• HF_HUB_OFFLINE=1: no HTTP calls, cached files only, an error if missing [doc]. Without it an uncached model is downloaded from Hugging Face [doc].

• --max-tokens defaults to 512 [doc]: a request without its own limit would stop mid-answer.

• Sampling defaults are greedy (--temp 0.0, --top-p 1.0) [doc]; set --temp/--top-p/--top-k from the model card's recommended settings.

• Ready log: Starting httpd at 127.0.0.1 on port P... [run].

• There is no context size to set: the cache grows with the conversation, up to the model's max_position_embeddings. Check that it is ≥ 65536 and that weights + 64K of cache fit before starting.