Coreweave GPU Cost Leak Hunter

CoreWeave GPU Cost Leak Hunter

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Audits a CoreWeave GPU cluster for real-dollar cost leaks — idle reserved capacity, GPUs on the wrong SKU, allocated-but-idle instances, and steady on-demand spend that should be committed — then emits a CFO-grokkable, dollar-ranked FinOps report.

Overview

CoreWeave ships no cost dashboard and no billing API (usage-monitoring docs). There is also no single "dollars" metric — spend is reconstructed by querying usage from CoreWeave's managed Grafana in PromQL and multiplying each resource's usage by its rate-card price. This skill does exactly that, then ranks the leaks by monthly dollar impact.

The math is deterministic: PromQL returns usage counts, and the bundled scripts/rank-and-report.py does every multiplication, sum, and ranking — the agent never eyeballs a number. Two of the four categories are billed waste (Confirmed); the other two are a right-sizing model (Estimated) and a commitment decision (At-risk), labeled so a CFO never reads a modeled number as recoverable cash. Deep domain knowledge lives in references/, loaded only when a leak needs it.

Prerequisites

• CoreWeave managed Grafana access — the Prometheus data source is reachable only to a member of the admin, metrics, or write group in the CoreWeave Cloud Console (usage-monitoring docs). This is the hard dependency; Step 1 probes it and fails fast if the group is missing.
• A Prometheus/Grafana query endpoint in $CW_PROM_URL (the Grafana data-source proxy, e.g. https://grafana.ORG.coreweave.com/api/datasources/proxy/uid/UID) and a bearer token in $CW_TOKEN for curl.
• kubeconfig for the cluster (CoreWeave-issued) so kubectl get can corroborate live GPU allocation and node labels.
• The rate card — CoreWeave publishes no price metric, so on-demand and committed rates are supplied to the ranker from references/gpu-right-sizing.md (dated snapshot of coreweave.com/pricing) or the customer's contract.
• jq and python3 for parsing query JSON and running the ranker.

Authentication. All auth comes from the environment ($CW_PROM_URL, $CW_TOKEN, $KUBECONFIG) — no secrets are hardcoded. Grafana enforces the group membership above on every query.

Instructions

The pipeline is detect → price → rank → report. PromQL returns usage; the dollar arithmetic runs in scripts/; deep knowledge loads from references/ on demand:

1. Verify metric access, fail fast if the group is missing.
2. Pull the 30-day spend baseline (usage × rate card).
3. Detect Leak 1 — idle reserved capacity (Confirmed).
4. Detect Leak 2 — wrong-GPU-type right-sizing waste (Estimated).
5. Detect Leak 3 — allocated-but-idle instances (Confirmed).
6. Detect Leak 4 — on-demand spend that should be committed (At-risk).
7. Rank by monthly dollar impact and render the CFO report.

Step 1: Verify Metric Access (fail fast, not mid-flow)

Probe billing:instance:total before anything else. An HTTP 401/403 or empty result means the token's principal is not in admin/metrics/write — STOP and report it; do not continue into the scans.

curl -sS -H "Authorization: Bearer $CW_TOKEN" \
  --data-urlencode 'query=count(billing:instance:total)' \
  "$CW_PROM_URL/api/v1/query" | jq -r '.status, (.data.result | length)'

If status is not success with a non-empty result, load references/promql-billing-setup.md and report the missing group access verbatim. Stop here.

Step 2: Pull the Spend Baseline

Reconstruct 30-day GPU node-hours per instance type. CoreWeave has no dollars metric, so this returns usage — the ranker multiplies by the rate card. Write the JSON to the working dir for the ranker.

curl -sS -H "Authorization: Bearer $CW_TOKEN" \
  --data-urlencode 'query=sum by (instance_type) (sum_over_time(billing:instance:total[30d:1h]))' \
  "$CW_PROM_URL/api/v1/query" > "$OUT/baseline.json"

The rate card and the per-category PromQL live in references/gpu-cost-leak-categories.md. Load it now — the four scans below reference its recording-rule notes.

Step 3: Leak 1 — Idle Reserved Capacity (Confirmed)

Reserved GPUs bill at the committed rate whether used or not. A reserved GPU sitting below a utilization floor is confirmed waste — you paid for it and it did no work. Cross reserved allocation (billing_gpu, filtered by the reservation label) against SM-active from DCGM.

curl -sS -H "Authorization: Bearer $CW_TOKEN" --data-urlencode \
  'query=sum by (instance_type,node) (avg_over_time(billing_gpu{reservation!=""}[30d:1h]))
     and on(node) (avg by (node) (avg_over_time(DCGM_FI_PROF_SM_ACTIVE[30d:1h])) < 0.05)' \
  "$CW_PROM_URL/api/v1/query" > "$OUT/leak1-idle-reserved.json"

The reservation label key is provider-specific — confirm yours with kubectl get nodes --show-labels. Waste = idle reserved GPU-hours × committed rate (ranker input).

Step 4: Leak 2 — Wrong-GPU-Type Right-Sizing (Estimated)

H100/H200 running small-model (~7B–30B) inference is over-paying: for that regime L40S is cheaper per token (directional — see gpu-right-sizing.md). Flag those instance-hours; the ranker re-prices them at the L40S rate.

curl -sS -H "Authorization: Bearer $CW_TOKEN" --data-urlencode \
  'query=sum by (instance_type) (sum_over_time(billing:instance:total{instance_type=~".*(h100|h200).*"}[30d:1h]))' \
  "$CW_PROM_URL/api/v1/query" > "$OUT/leak2-wrong-gpu.json"

This is Estimated: the rate delta is exact rate-card math, but throughput equivalence on L40S is a model. Confirm the served model size with the cluster owner before acting; FP8 serving needs Hopper/Ada, not Ampere (see gpu-right-sizing.md).

Step 5: Leak 3 — Allocated-but-Idle Instances (Confirmed)

On-demand GPUs that are allocated (billing) but running at low SM-utilization / low MFU bill the full on-demand rate for no work — confirmed billed waste, the GPU twin of an idle cluster.

curl -sS -H "Authorization: Bearer $CW_TOKEN" --data-urlencode \
  'query=(avg by (node,instance_type) (avg_over_time(DCGM_FI_PROF_SM_ACTIVE[30d:1h])) < 0.05)
     and on(node) (sum by (node) (avg_over_time(billing_gpu{reservation=""}[30d:1h])) > 0)' \
  "$CW_PROM_URL/api/v1/query" > "$OUT/leak3-idle-ondemand.json"

Corroborate with kubectl get pods -A --field-selector=status.phase=Running to confirm nothing is actually scheduled on the flagged node. Waste = idle on-demand GPU-hours × on-demand rate.