Ops Query

Query

Combine on-chain indexer data, off-chain broker telemetry, and CloudWatch service logs to investigate operational issues and find insights.

Setup

Set up the data sources needed for the investigation. Not all sources are needed for every query -- use the ones relevant to the task.

1. Read network_secrets.toml from the repo root. If it exists, it contains credentials for all environments (indexer API keys, telemetry DB URLs/passwords, AWS creds). Also read network_address_labels.json (same directory) for labelling addresses -- it is plain JSON ({"0xaddr": "label", ...}). If network_secrets.toml is not present, recommend the user create it -- instructions and credentials are in the Boundless runbook. If network_address_labels.json is not present, recommend the user create it -- the canonical address mapping is in the Boundless runbook.

2. Read and follow the ops-indexer-query skill at .claude/skills/ops-indexer-query/SKILL.md to set up indexer access (MARKET_INDEXER_URL, ZKC_INDEXER_URL, optional INDEXER_API_KEY, and the indexer_get helper function).

3. Read and follow the ops-telemetry-query skill at .claude/skills/ops-telemetry-query/SKILL.md to set up Redshift access (REDSHIFT_URL). Before writing any telemetry SQL, read crates/boundless-market/src/telemetry.rs for exact column names and enum values.

4. Read and follow the ops-logs-query skill at .claude/skills/ops-logs-query/SKILL.md to set up CloudWatch log access (AWS credentials, log group discovery). Use when the investigation benefits from raw service logs -- especially for our own operated provers, or for other services like indexer, order stream, slasher, etc.

5. Ask the user which network they want to investigate (determines which market indexer + ZKC indexer to use).

Data Source Overview

The key join between indexer and telemetry is request_id and request_digest. The indexer's lock_prover_address or fulfill_prover_address corresponds to telemetry's broker_address. Logs can be correlated by searching for the same request_id or request_digest within the relevant time window.

Telemetry is opt-in -- not all brokers send telemetry. If a prover address has no telemetry data, note this to the user. CloudWatch logs are only available for services we operate.

Investigation Workflow

All investigations follow the same pattern:

1. Identify targets -- Use the indexer to find the relevant requests/addresses/time periods.
2. Correlate with telemetry -- Use the request IDs, digests, or broker address + time window to look up telemetry data. Telemetry provides a pre-processed view with structured skip reasons, error codes, proving durations, and estimation accuracy -- this should answer most questions about why orders were skipped, failed, or slashed without needing raw logs.
3. Dig into logs (last resort) -- Only go to CloudWatch logs if the indexer and telemetry data are insufficient. Logs are useful when you need raw error messages, stack traces, or runtime details that telemetry doesn't capture (e.g. infrastructure-level failures, panics, or non-prover service issues). One particularly useful check: look for recent deployments in bento prover logs. Nightly deployments restart Docker Compose and can explain gaps in telemetry, sudden behavior changes, or outages. New code deployed can also introduce bugs. See the "Checking for Recent Deployments" section in the ops-logs-query skill.
4. Analyze and synthesize -- Combine findings from all sources into a coherent narrative.

Rate limit all sources:

• Indexer: sleep 1 between requests, sleep 2 between pagination pages.
• Redshift: No rate limit, but use LIMIT on exploratory queries.
• CloudWatch: sleep 1 between paginated log queries.

Pre-Built Investigations

Before starting any work, check if the user's question matches a pre-built investigation. These are tested playbooks with the right queries, presentation format, and step-by-step instructions. Using them produces consistent, comprehensive results. Each lives in its own file under references/:

If the user's question clearly maps to one of these, read the corresponding file and follow it step by step. If it doesn't fit any pre-built investigation, fall back to the general Investigation Workflow above and build a custom investigation.

Aurora DB Instances

When investigating RDS/Aurora issues (storage, CPU, connections, etc.), never assume the DB identifier reflects the actual instance role. Instance identifiers containing reader or writer may be mislabeled -- the name is set at creation time and does not update if Aurora promotes/demotes instances.

Always determine the actual role by querying the instance metadata:

aws rds describe-db-instances \
  --query 'DBInstances[?contains(DBInstanceIdentifier, `prod-8453-indexer`)].{id: DBInstanceIdentifier, role: DBInstanceArn}' \
  --output table

Or more directly, check the cluster's member list with roles:

aws rds describe-db-clusters \
  --db-cluster-identifier "CLUSTER_ID" \
  --query 'DBClusters[0].DBClusterMembers[].{id: DBInstanceIdentifier, isWriter: IsClusterWriter}' \
  --output table

Use IsClusterWriter: true/false as the source of truth. When reporting findings, always state the actual role alongside the identifier, e.g. "instance *-reader-v19 (actual role: writer)".