Ar Io Gateway Operator

AR.IO Gateway Operator (generic)

You are operating an AR.IO node — an Arweave gateway that indexes, caches, serves, and signs blockchain data. Deployments share the same architecture but differ in domains, paths, and which sidecars are present. Run scripts/health-check first every session: it prints a one-screen snapshot you can compare against the prior state before deciding to act.

If a deployment-specific overlay skill is loaded (e.g. vilenarios-gateway), it provides the local repo paths, domains, and which sidecars exist. Treat the overlay as authoritative for those facts.

Run first: scripts/health-check

# from the ar-io-node repo root
.claude/skills/ar-io-gateway-operator/scripts/health-check

What it checks and why:

The script only covers the core ar-io-node containers defined in this repo's docker-compose. Sidecar checks belong in a deployment-overlay skill that calls this script then adds its own checks on top.

How the gateway works (the model an operator needs)

Knowing the data flows tells you where to look when symptoms appear. Source of truth is src/system.ts (DI wiring) and docs/; this section captures the operationally-load-bearing facts.

Request flow

client → (optional edge: nginx/CDN) → envoy :3000 → core :4000 → response with X-AR-IO-* trust headers
                                                              ↘ webhook fan-out → optional sidecars

Every served data response carries trust headers (X-AR-IO-Verified, X-AR-IO-Trusted, X-AR-IO-Stable, X-AR-IO-Hops, plus X-Arweave-* tag passthrough). Those headers are what HTTPSig signs — see src/middleware/httpsig.ts TRIGGER_HEADERS. Endpoints with no trust trigger (/ar-io/info, /ar-io/healthcheck, /graphql errors) are deliberately not signed.

Data retrieval — composite source with fallback chain

Configurable via .env. Defaults:

• ON_DEMAND_RETRIEVAL_ORDER=trusted-gateways,ar-io-network,chunks-offset-aware,tx-data — what /raw/<id> and /<id> walk through on a cache miss
• BACKGROUND_RETRIEVAL_ORDER=chunks — used by the unbundler when fetching bundle bodies

Each source's failures are logged with class so you can grep (GatewaysDataSource, TxChunksDataSource, ChunksDataSource, etc.). Persistent failures from one tier are normal — the chain is designed to fall through. Look for Failed to fetch chunk data from any source / all sources failed as the real failure signal, not individual tier warnings.

Three response headers worth knowing: X-Cache: HIT|MISS (gateway-side contiguous cache), X-AR-IO-Hops (how many AR.IO peers were traversed before this gateway served), X-AR-IO-Trusted / X-AR-IO-Verified (whether the data has been hash-verified against the chain).

Indexing pipeline

Two write paths run side-by-side:

1. SQLite primary indexer (always on) — BlockImporter worker imports blocks one at a time into new_blocks/new_transactions (the unstable head, ~last 18 blocks), and a separate process promotes them to stable_blocks/stable_transactions once they're past the reorg horizon. ANS-104 bundle unbundling is gated by ANS104_UNBUNDLE_FILTER (often narrow). Database access is in a worker thread (StandaloneSqlite) — never call SQLite synchronously from the main thread.

2. ClickHouse pipeline (optional, --profile clickhouse) — Stable rows are exported to parquet then imported by clickhouse-auto-import (hourly cycles) into staging_* and finally transactions. Streaming pipeline (PR #699 / clickhouse-streaming-unstable-head branch) mirrors the unstable-head SQLite writes directly into ClickHouse new_blocks / new_transactions so GraphQL can query the head without waiting for the parquet round-trip. Activated by CLICKHOUSE_STREAMING_ENABLED=true. Reorgs trigger bounded ALTER TABLE ... DELETE WHERE on the new_* tables.

GraphQL routing

/graphql reads from up to three legs that get merged: stable transactions table in CH, streaming new_transactions in CH, and the SQLite new_* fallback. With CLICKHOUSE_GQL_SKIP_SQLITE_READS=true, the SQLite leg is dropped to a tight-timeout fallback only. With GRAPHQL_ON_DEMAND_RESOLUTION_ENABLED=true (default), transaction(id) queries that miss the local DB will fall back to extracting from ANS-104 bundle binaries on demand — single-ID only, capped by GRAPHQL_ON_DEMAND_RESOLUTION_MAX_CONCURRENT.

ANS-104 unbundling pipeline

Whenever a bundle TX matches ANS104_UNBUNDLE_FILTER, it walks a four-queue chain. Knowing the chain is essential for diagnosing throughput regressions — a single queue at cap can mask everything downstream of it.

bundleDataImporter ─→ ans104Unbundler ─→ ans104DataIndexer ─→ dataItemIndexer ─→ SQLite writer
   download workers      parser workers     post-parse buffer     pre-write buffer    single-writer
   (ANS104_DOWNLOAD_…)   (ANS104_UNBUNDLE_)  (ANS104_DATA_…_SIZE)  (DATA_ITEM_…_SIZE)

Tune knobs (all env-overridable):

The buffer between soft gate and hard cap is where data loss happens. With defaults the gate engages at 100 K but the hard cap is 500 K — a 5× buffer. Under heavy load, in-flight parser workers can flood items into the queue after the gate engages but before the indexer drains, exceeding the hard cap and silently dropping items. Symptom: data_items_dropped_total{queue_name="dataItemIndexer"} climbs while bundles_unbundled_total keeps increasing — bundles look "indexed" but their items aren't searchable.

Mitigations:

• Tighten MAX_DATA_ITEM_QUEUE_SIZE so the gate engages well before the hard cap (e.g., 25 000 with 500 000 hard caps, giving a 20× buffer).
• Raise the hard cap if memory permits (each item slot ≈ 1–2 KB).
• Don't add parser workers when data_items_dropped_total is non-zero — more workers = bigger floods past the gate.

Pipeline diagnostic: a single curl -sf .../ar-io/__gateway_metrics | grep -E 'queue_length|data_items_dropped|bundles_unbundle_in_flight' tells you which queue is the bottleneck.

Caching layers (in order, fastest first)

X-Cache: HIT on a /raw/<id> response means the gateway's contiguous data store served bytes without going to peers/Arweave. The edge layer is separate — nginx's X-Cache-Status: HIT is a different layer.

ArNS resolution

CompositeArNSResolver walks resolvers in order: TrustedGatewayArNSResolver (asks TRUSTED_ARNS_GATEWAY_URL, default https://__NAME__.turbo-gateway.com), then OnDemandArNSResolver (queries the on-chain ario-arns and ario-ant programs directly via SOLANA_RPC_URL). The base name set is paginated into an in-memory ArNSNamesCache at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to data path) or, on the streaming-head branch, IPFS CIDs (route to /ipfs/<cid> via the Kubo sidecar). Unknown names log Unable to resolve name against all resolvers — that's normal user-error traffic, not a service failure.

/<name> and /<name>/<path> requests carry X-ArNS-* trust headers in the response. Manifest path resolution still uses StreamingManifestPathResolver; the "from index" path is not implemented yet (logs warn not implemented then falls back to data-side resolution, which works).

Network identity, observer, and incentives

A gateway is a registered participant in the AR.IO network, not just a piece of software. Two distinct Solana identities matter:

• Operator — AR_IO_WALLET (pubkey, base58) plus a signing key as either SOLANA_KEYPAIR_PATH (JSON file) or SOLANA_PRIVATE_KEY (base58 secret, Phantom-export form). Stakes ARIO tokens (20,000 minimum = 20000000000 mARIO since v3.0.0), receives rewards, signs cranker instructions when ENABLE_EPOCH_CRANKING=true.
• Observer — OBSERVER_WALLET (pubkey) plus a signing key as either OBSERVER_KEYPAIR_PATH or OBSERVER_PRIVATE_KEY. Submits save_observations instructions and signs HTTPSig responses. The two roles are deliberately separable so the observer key can sit on the gateway box while the operator/staking key stays in cold storage. Setting both forms (path and PK string) for the same role is rejected at startup.

The Solana key designated for the observer role is the HTTPSig signing key. The observer's address is already registered in the on-chain Gateway Registry (GAR), so verifiers can confirm a signed response traces back to a staked, registered gateway with a single GAR lookup. No attestation document is uploaded to Arweave on the Solana path — the chain is the source of truth.

Four Solana program IDs configure which network the gateway talks to: ARIO_CORE_PROGRAM_ID, ARIO_GAR_PROGRAM_ID, ARIO_ARNS_PROGRAM_ID, ARIO_ANT_PROGRAM_ID. Plus SOLANA_RPC_URL (premium provider strongly recommended in production; public mainnet-beta throttles hard). GET /ar-io/info echoes the resolved programIds object, which is the fastest way to confirm a running gateway is pointed at the network you think it is.

Observer service knobs:

• RUN_OBSERVER=true (default) — runs the observer alongside the gateway
• SUBMIT_CONTRACT_INTERACTIONS=true (default) — submits generated reports on-chain
• OBSERVATIONS_PER_GATEWAY (default 3) — how many times each gateway is observed per epoch (majority vote)
• OBSERVATION_WINDOW_FRACTION (default 0.5) — what fraction of the daily epoch window observations are spread across
• REFERENCE_GATEWAY_MIN_CONSECUTIVE_PASSES / REFERENCE_GATEWAY_MIN_EPOCH_COUNT — eligibility thresholds for using a network gateway as a reference

Epoch cadence is daily. Observation reports drive reward eligibility. Operators may themselves be selected as observers for other gateways.

Network onboarding (operator side, only needed once):

• npm install -g @ar.io/sdk — provides the ar.io CLI binary. Use ar.io get-gateway --address <pubkey> to read state, ar.io join-network ... to register.
• The CLI accepts the operator key as either --wallet-file <path> (JSON) or --private-key '<base58>' (Phantom export — bs58.decode runs in-process).
• Every Solana command needs -t solana and the four program IDs (--core-program-id / --gar-program-id / --arns-program-id / --ant-program-id) plus --rpc-url. As of @ar.io/sdk@^4.0.0-solana.22 the bundled "mainnet" constants are still placeholders, so passing --mainnet alone resolves to non-existent addresses — always pass the explicit program IDs the network team publishes for whichever environment you're targeting (mainnet, staging-devnet, local devnet).
• Register via the CLI or the gateways.ar.io portal: operator wallet, observer wallet, FQDN, Properties Tx ID, label, stake (--operator-stake 20000 in whole ARIO — the CLI converts to mARIO internally via requiredMARIOFromOptions → new ARIOToken(...).toMARIO()).
• Gateway must be reachable at the registered FQDN with valid TLS, AND configured for the same programIds set you joined with. The usual cause of a "I joined but nothing's working" report is a gateway whose .env still pins the old network's program IDs.
• Optional delegated staking — operators can accept delegations with a configurable reward share. --min-delegated-stake is mARIO directly (no ARIO→mARIO conversion); --operator-stake is the only stake-input flag that takes whole ARIO.

Serving content at the root path

Two mutually-relevant env vars decide what / serves:

• APEX_TX_ID — pin a specific Arweave TX as the apex. Cache-Control bounded by CACHE_APEX_MAX_AGE with must-revalidate so apex rotations propagate.
• APEX_ARNS_NAME — resolve an ArNS name as the apex. Comma-separated values position-map to ARNS_ROOT_HOST entries when there's more than one host.
• ARNS_ROOT_HOST — the gateway's primary domain(s) for ArNS subdomain resolution.

Index sharing sidecar (optional)

The index-swarm sidecar (compose profile index-swarm) publishes this gateway's CDB64 root-TX index bands, signed with the observer key, and subscribes to other gateways' bands, installing them under data/indexes/installed/ where the gateway loads them without a restart. It resolves publishers through the gateway's own /ar-io/peers, so it makes no RPC calls. The gateway serves /ar-io/indexes (the signed document) and the byte routes, metered like data. The optional torrent engine (profile index-swarm-torrent; setting INDEX_SWARM_ENGINE_AUTH turns it on) adds a BitTorrent tier: peers first, the HTTP routes as fallback, every subscriber seeding what it installs. The engine runs on its own Docker network and filters private addresses; on a LAN-only swarm set INDEX_SWARM_ENGINE_BLOCK_PRIVATE=false. Health: index_subscription_manifest_age_seconds (publisher gone quiet), index_swarm_installed_bands, index_subscription_bytes_total{transport}, index_swarm_engine_available. On a publisher, who shares: index_swarm_tracker_seeding_hosts (other hosts seeding a band) and index_swarm_tracker_seeders. Full guide: docs/index-swarm.md.

A fleet behind a load balancer (docs/index-swarm.md, "Publishing torrents from a fleet"): only the node /ar-io/indexes* is pinned to publishes and runs the engine and tracker; its peer port goes to its own public IP, not through the LB (set INDEX_SWARM_ENGINE_PUBLIC_HOST), and the tracker is direct or /announce proxied with INDEX_SWARM_TRACKER_TRUSTED_PROXIES. Rate limiter and x402 need nothing new: byte routes and the WebSeed are metered, the document and .torrent are free. Other nodes that answer lookups subscribe over HTTP with url pointed at the publishing node, allowlisted in its RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST.

Ports and disk. The engine's peer port (INDEX_SWARM_ENGINE_PORT, 6881 TCP+UDP) and the tracker's (INDEX_SWARM_TRACKER_PORT, 6969 TCP) are Docker-published, so ufw neither blocks nor protects them (filter where Docker forwards: the DOCKER-USER chain with Docker's default iptables backend; with its nftables backend, which has no DOCKER-USER, a chain in your own table on the forward hook at a priority before Docker's), and a port already held on the host stops the container starting. A publisher needs its peer port reachable, and the tracker port only when its torrents announce to it directly (INDEX_SWARM_TRACKERS naming this node's port): not when /announce is proxied through a load balancer, nor when it publishes with no tracker (it warns, and peers then meet only through DHT and the WebSeed). A subscriber needs neither (behind NAT it still downloads and seeds to peers it dials), and index-swarm-status reports a closed port as WARN, not FAIL. The engine has UPnP off. INDEX_SWARM_MAX_DISK_BYTES (50 GiB from the setup script) is a ceiling, not a reservation: nothing checks free space, so a full disk under data/indexes is the operator's to prevent. That directory must be one filesystem (renames, hard links); move it whole with INDEX_SWARM_DATA_PATH. Upload defaults to 10 MB/s and 100 GB a UTC day, up to about 3 TB a month.

Build and check bands with ./tools/ar-io-node (docs/cli.md, "For scripts and agents"): index-band-build (CSV records in, header check against --gateway-url, published into data/indexes/published/root-tx-index) and index-band-verify. JSON on stdout, exit 1 on failure, safe to rerun (an identical band reports "unchanged": true). Point --gateway-url at a gateway that holds the roots (https://turbo-gateway.com); a gateway that must fetch them times out and the band is refused as under-checked, not wrong. Any other command runs as the SDK's ar.io CLI.

Bands are built by the index-export service (compose profile index-export; docs/index-swarm.md, "Producing bands"), once a day at INDEX_EXPORT_RUN_AT_UTC from INDEX_EXPORT_SOURCES (this gateway's ClickHouse, else SQLite; peers and a bundler overlay merge). It needs INDEX_EXPORT_START_HEIGHT for its first run and INDEX_EXPORT_HEADER_CHECK_URL always (a gateway holding the roots; not core:4000 on a gateway that fetches them). Dry-run before starting: docker compose --profile index-export run --rm index-export --once --dry-run. Bring it up by name with --no-deps. Its /healthz is loop liveness only; outcomes are index_export_runs_total{result,reason} and index_export_last_success_timestamp_seconds{kind="d"} (alert past 2 days). couldnt_check retries itself (15 min doubling to 2 h); rejected (a wrong header, conflicts over 1%) never does and needs a look. A stale data/indexes/export/lock (5 min untouched) is cleared by the next run; state.json there is re-derivable except adoptions. With parquet-l1 in INDEX_EXPORT_KINDS it also builds L1 bands from core.db, and adds lookup files (layout l1-3) to published L1 bands in place, keeping their ids (index_export_runs_total{kind="derive"}). Upgrade a gateway that subscribes to parquet-l1 before its publisher writes l1-3 (an older one refuses those bands, naming the layout, and keeps its copies), and a publisher's index-swarm with or before its index-export (an older sidecar can't describe an l1-3 band, so it goes unpublished).

Set up and check with the scripts, not by hand: ./tools/index-swarm-setup --subscribe <publisher wallet> --torrent --dry-run shows the .env changes (subscription, CDB64 sources, lookup order, engine password); --restart applies them, recreating only what needs it with the running gateway's compose files. ./tools/index-swarm-status is the first thing to run when asked whether index sharing works: read-only, ok/WARN/FAIL per check with the fix, exit 1 on a failure.

Filters and webhooks (the gateway's contract with sidecars)

Three filters compose JSON expressions (tags, attributes, or/and/always/never) — see docs/filters.md:

• ANS104_UNBUNDLE_FILTER — which bundles get unbundled
• ANS104_INDEX_FILTER — which data items inside matched bundles get persisted
• WEBHOOK_INDEX_FILTER — which indexed items trigger webhook emission

Webhook events fired: data-cached, tx-indexed, ans104-data-item-indexed. WEBHOOK_TARGET_SERVERS is a comma-separated list of URLs that receive POSTs. If you change a filter to broaden indexing, expect proportionally higher webhook volume.

Common operations

Bring up

# from the ar-io-node repo root
docker compose --profile clickhouse up -d

The --profile clickhouse flag matters: without it, ClickHouse and the auto-import container stay down and the streaming pipeline goes silent. Other profiles (grafana, bundler, litestream, ao, hb) gate their own services.

Wait for ready:

until curl -sf http://localhost:4000/ar-io/healthcheck >/dev/null; do sleep 2; done

Rebuild from local source (after dev-branch code changes)

docker compose --profile clickhouse build core clickhouse-auto-import
docker compose --profile clickhouse up -d --force-recreate core clickhouse-auto-import

--force-recreate is required — without it, compose sees the same image name (:latest) and skips recreation even though the image digest changed. Confirm with docker inspect <container> --format '{{.Image}}' — the SHA must differ from before.