Ops Logs Query

Logs Query

Query AWS CloudWatch Logs for Boundless services on prod/staging.

Prerequisites

1. Read network_secrets.toml from the repo root. Extract the AWS credentials for the target environment from [aws.prod] or [aws.staging] (access_key_id, secret_access_key). If the file is not present, recommend the user create it -- instructions and credentials are in the Boundless runbook.

2. Export credentials before running any queries:

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-west-2"

Finding Log Groups

Prover log groups

We run one prover, and it is prod-only. Its logs go to the CloudWatch log group boundless/hlc/prover -- note there is no leading slash -- in the log stream prover-prod.

There is no staging prover. A same-named group exists in the staging account (stream prover-staging) but it has been idle since 2026-06-22. If asked about a staging prover, say it is not deployed rather than going looking for its logs.

Retired: the Bento prover log groups

The /boundless/bento/prover-* groups and the *-bento-prover-* groups are the decommissioned Bento fleet. Do not query them.

They still exist, so a query against them succeeds and returns zero events. That reads as "no errors found" and is a false negative -- it says nothing about prover health. Last events were May 2026 for /boundless/bento/prover-* and January 2026 for *-bento-prover-*/bento.

The one still-live group under the bento prefix is the explorer, /boundless/bento/explorer-8453-prod-release.

⚠️ infra/cw-monitoring/Pulumi.production.yaml and Pulumi.staging.yaml still list the retired Bento hostnames (proverType: bento) and have no entry for boundless/hlc/prover. Do not treat them as the source of truth for prover log groups until they are updated.

We only have prover log groups for provers we operate. External provers do not have queryable logs.

Provers we operate are labeled with a BP prefix in network_address_labels.json (e.g. BP1, BP2, BPNightlyAWS). When investigating any issue, always highlight what our BP provers are doing -- did they skip, fail, drop, or fulfill? This should be called out explicitly even when the investigation is not specifically about our provers.

Discovering log groups for other services

Other services (slasher, distributor, order stream, order generator, indexer backend, indexer API, etc.) have log groups that follow a naming convention but may change. Discover them dynamically rather than hardcoding.

Log group naming convention: l-<staging|prod>-<chain_id>-<service-name>-<chain_id>-<resource>

Example: l-staging-167000-indexer-api-167000-lambda

Known chain IDs:

• 84532 = Base Sepolia
• 8453 = Base Mainnet
• 167000 = Taiko Mainnet
• 11155111 = Eth Sepolia
• 1 = Eth Mainnet

To find log groups for a specific service, search by prefix. Use the environment (staging/prod) and optionally the chain ID and service name:

aws logs describe-log-groups \
  --log-group-name-prefix "l-staging-84532" \
  --query 'logGroups[].logGroupName' --output table

To find log groups for a specific service across all chains in an environment:

aws logs describe-log-groups \
  --query 'logGroups[?contains(logGroupName, `staging`) && contains(logGroupName, `indexer`)].logGroupName' \
  --output table

To list all log groups in an environment:

aws logs describe-log-groups \
  --log-group-name-prefix "l-staging" \
  --query 'logGroups[].logGroupName' --output table

Also check the prover log group:

aws logs describe-log-groups \
  --log-group-name-prefix "boundless/hlc/prover" \
  --query 'logGroups[].logGroupName' --output table

Some services have multiple log groups for different components (e.g. an indexer may have separate groups for the backend worker and the API lambda). When investigating an issue, check all matching log groups.

Service name patterns

Common service name fragments to search for:

Querying Logs

Always filter by time range. Log groups are high-volume and queries without time bounds will be slow or hit limits.

Use aws logs filter-log-events for searching. Key parameters:

• --log-group-name: required
• --start-time / --end-time: Unix milliseconds (required -- always set these)
• --filter-pattern: CloudWatch filter syntax for searching log content
• --output json: pipe through jq for readability

Computing timestamps

Convert human-readable times to Unix milliseconds:

START=$(date -j -u -f "%Y-%m-%dT%H:%M:%SZ" "2026-03-30T00:00:00Z" +%s 2>/dev/null || date -d "2026-03-30T00:00:00Z" +%s)
START_MS=$((START * 1000))

END=$(date -j -u -f "%Y-%m-%dT%H:%M:%SZ" "2026-03-31T00:00:00Z" +%s 2>/dev/null || date -d "2026-03-31T00:00:00Z" +%s)
END_MS=$((END * 1000))

For relative times:

NOW_MS=$(date +%s)000
ONE_HOUR_AGO_MS=$(( ($(date +%s) - 3600) * 1000 ))
SIX_HOURS_AGO_MS=$(( ($(date +%s) - 21600) * 1000 ))
ONE_DAY_AGO_MS=$(( ($(date +%s) - 86400) * 1000 ))

Searching by request ID

The most common query pattern. Request IDs appear in log messages as hex values (e.g. 0x2a):

aws logs filter-log-events \
  --log-group-name "$LOG_GROUP" \
  --start-time "$ONE_HOUR_AGO_MS" \
  --end-time "$NOW_MS" \
  --filter-pattern '"0xREQUEST_ID"' \
  --output json | jq '.events[] | {timestamp: (.timestamp / 1000 | todate), message: .message}'

Searching by request digest

aws logs filter-log-events \
  --log-group-name "$LOG_GROUP" \
  --start-time "$ONE_HOUR_AGO_MS" \
  --end-time "$NOW_MS" \
  --filter-pattern '"0xDIGEST"' \
  --output json | jq '.events[] | {timestamp: (.timestamp / 1000 | todate), message: .message}'

Searching for errors

aws logs filter-log-events \
  --log-group-name "$LOG_GROUP" \
  --start-time "$ONE_HOUR_AGO_MS" \
  --end-time "$NOW_MS" \
  --filter-pattern '"ERROR"' \
  --output json | jq '.events[] | {timestamp: (.timestamp / 1000 | todate), message: .message}'

Searching across multiple log groups

When a service has multiple log groups, query each one:

for LG in "l-staging-84532-indexer-api-84532-lambda" "l-staging-84532-market-indexer-84532-task"; do
  echo "=== $LG ==="
  aws logs filter-log-events \
    --log-group-name "$LG" \
    --start-time "$ONE_HOUR_AGO_MS" \
    --end-time "$NOW_MS" \
    --filter-pattern '"ERROR"' \
    --output json | jq '.events[] | {timestamp: (.timestamp / 1000 | todate), message: .message}'
  sleep 1
done

Pagination

filter-log-events returns a nextToken when there are more results:

TOKEN=""
while true; do
  if [ -n "$TOKEN" ]; then
    RESP=$(aws logs filter-log-events \
      --log-group-name "$LOG_GROUP" \
      --start-time "$START_MS" \
      --end-time "$END_MS" \
      --filter-pattern '"0xREQUEST_ID"' \
      --next-token "$TOKEN" \
      --output json)
  else
    RESP=$(aws logs filter-log-events \
      --log-group-name "$LOG_GROUP" \
      --start-time "$START_MS" \
      --end-time "$END_MS" \
      --filter-pattern '"0xREQUEST_ID"' \
      --output json)
  fi

  echo "$RESP" | jq '.events[] | {timestamp: (.timestamp / 1000 | todate), message: .message}'
  TOKEN=$(echo "$RESP" | jq -r '.nextToken // empty')
  [ -z "$TOKEN" ] && break
  sleep 1
done

CloudWatch Filter Pattern Syntax

• "exact phrase" -- matches logs containing the exact phrase (quotes required)
• ?term1 ?term2 -- OR: matches logs containing either term
• "term1" "term2" -- AND: matches logs containing both terms
• "ERROR" "request_id" -- combine filters