Setup Prover

Deploy Prover

Interactive helper for deploying the Boundless prover stack to GPU servers.

What you can help with

1. New prover deployment — create inventory, deploy full stack (NVIDIA + Docker + bento)
2. Redeploy/update — update images, config, or git version on an existing server
3. Tear down — stop services and clean up
4. Troubleshoot — check service health, logs, GPU status
5. Configure — tune broker.toml settings (peak_prove_khz, concurrency, etc.)

Prerequisites

Before deploying, verify the user has:

1. Ansible — ansible --version (install: brew install ansible)
2. SSH access — ssh <user>@<host> "echo connected"
3. GPU server — Ubuntu 22.04 or 24.04 with NVIDIA GPU hardware

Deployment workflow

1. Create inventory file

Create ansible/inventory-<name>.yml with the server details. NEVER hardcode secrets — use env var lookups:

---
all:
  vars:
    ansible_user: ubuntu
    ansible_ssh_private_key_file: ~/.ssh/<key>
    # ansible_become_password: "{{ lookup('env', 'ANSIBLE_BECOME_PASSWORD') }}"  # uncomment if server needs sudo password
    prover_version: main  # or a specific branch/tag
  children:
    <group_name>:
      hosts:
        <ip_address>:
          prover_postgres_password: "<password>"
          prover_minio_root_pass: "<password>"
          prover_image_tag: "nightly-latest"
          prover_boundless_build: ""
          prover_rpc_urls: "{{ lookup('env', 'PROVER_RPC_URLS') }}"
          prover_private_key: "{{ lookup('env', 'PROVER_PRIVATE_KEY') }}"
          # prover_executor_count and max_concurrent_preflights default to 8 (matching broker-template.toml)
          prover_broker_toml_url: "https://raw.githubusercontent.com/boundless-xyz/boundless/refs/heads/main/broker-template.toml"
          # prover_broker_toml_local: "/path/to/custom/broker.toml"  # uncomment to use a local file instead

Ensure the inventory file is in .gitignore (ansible/inventory-*.yml pattern).

2. Set environment variables

The inventory uses {{ lookup('env', 'VAR') }} for secrets. These resolve on the machine running ansible-playbook, not on the remote server. They must be exported in the same shell that runs the playbook.

export PROVER_PRIVATE_KEY="<broker private key>"
export PROVER_RPC_URL_8453="<Base RPC URL>"
export PROVER_RPC_URL_167000="<Taiko RPC URL>"
ansible-playbook -i inventory-<name>.yml prover.yml  # must be in the same shell

3. Test connectivity

cd ansible
ansible all -i inventory-<name>.yml -m ping

4. Deploy

PROVER_PRIVATE_KEY="$PROVER_PRIVATE_KEY" \
PROVER_RPC_URLS="$PROVER_RPC_URLS" \
ansible-playbook -i inventory-<name>.yml prover.yml

This installs: NVIDIA drivers, Docker + NVIDIA Container Toolkit, then deploys the bento compose stack.

5. Verify

# Check all services are running
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env ps"

# Check GPU is detected
ssh <user>@<host> "nvidia-smi --query-gpu=name,memory.total --format=csv,noheader"

# Check broker logs
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env logs broker --tail 20"

# Check all images are correct
ssh <user>@<host> "sudo docker compose -f /opt/bento/compose.yml --env-file /opt/bento/.env ps --format '{{.Name}}\t{{.Image}}'"

Image modes

Before generating the inventory, ask the user which image mode they want. This matters because building from source is slow (Rust compilation) but required when deploying code from a feature branch that doesn't have CI-built images.

Ask: "Do you want to use pre-built images or build from source? If building from source, which components?"

prover_image_tag takes precedence over prover_image_version. When set, ALL images use the exact same tag.

Selective build + pre-built (hybrid mode)

You can combine prover_image_tag with prover_boundless_build to build only specific services from source while pulling pre-built images for everything else. The justfile handles this by clearing the image tag for the built service at runtime (export BROKER_IMAGE="") so docker compose builds it from the local Dockerfile instead of pulling.

Example: deploy a feature branch that only changes the broker:

prover_image_tag: "nightly-latest"       # pre-built images for agents, rest_api, etc.
prover_boundless_build: "broker"          # build only the broker from the branch source

Valid prover_boundless_build values: "all", "broker", "rest_api", "exec_agent", "aux_agent", "gpu_prove_agent", "miner", or space-separated combinations (e.g. "broker rest_api").

Broker configuration

Ask the user: "Do you want to use an existing broker config from the repo, or specify your own config directory?"

Option A: Use an existing config. List the directories in ansible/roles/prover/configs/broker/ and let the user pick:

prover_broker_config_dir: "roles/prover/configs/broker/<chosen-config>"

Option B: Specify a custom directory. The user provides a path to a directory containing their own config files:

prover_broker_config_dir: "/path/to/my/broker-config"

Ansible copies the entire directory contents to /opt/bento/ on the remote server.

The config directory should contain:

• broker.toml — base config shared across all chains
• broker.{chain_id}.toml — per-chain overrides (e.g. broker.8453.toml, broker.167000.toml)

Per-chain override files must go in a chain-overrides/ subdirectory (e.g. chain-overrides/broker.8453.toml). This is the directory mounted into the Docker container.

IMPORTANT: broker.toml must include min_mcycle_price and max_collateral even if per-chain overrides replace them. These are required fields — the broker fails to parse the base config without them.

Fallback options (when prover_broker_config_dir is not set):

• prover_broker_toml_local — copies a single local file to /opt/bento/broker.toml
• prover_broker_toml_url — downloads from a URL (default: broker-template.toml from main branch)

Auto-generating config with the CLI wizard

The boundless CLI has a wizard that generates optimized configs:

# Run locally — auto-detects local hardware, or you can manually enter the server's specs
boundless prover generate-config --broker-toml-file ./broker-<host>.toml

# Then use it in the inventory:
# prover_broker_toml_local: "./broker-<host>.toml"

The wizard calculates peak_prove_khz, max_concurrent_preflights, max_exec_agents, and other values. If run on a machine without GPUs (e.g. your Mac), it will prompt you to enter the values manually.

Key settings to tune per GPU:

Selective deployment

Use --skip-tags to skip specific roles:

# Skip NVIDIA driver installation
ansible-playbook -i inventory-<name>.yml prover.yml --skip-tags nvidia

# Skip both NVIDIA and Docker (host already has them)
ansible-playbook -i inventory-<name>.yml prover.yml --skip-tags nvidia,docker

Note: --tags prover does NOT work as expected — the individual tasks inside the role are not tagged. Use --skip-tags instead. IMPORTANT: --skip-tags nvidia,docker will ALSO skip the prover role because it is tagged with both prover and docker. To skip only the NVIDIA and Docker install roles, use --skip-tags nvidia-install,docker-install. The NVIDIA/Docker roles are idempotent so running the full playbook on an already-configured host is safe and fast.