Gumroad Prod Console

Gumroad production console

Read-only Rails runner against the production read replica via bastion SSH.

Execution

.claude/skills/gumroad-prod-console/scripts/prod_query.sh 'puts User.count'
.claude/skills/gumroad-prod-console/scripts/prod_query.sh /tmp/query.rb
echo 'puts User.count' | .claude/skills/gumroad-prod-console/scripts/prod_query.sh

Multi-line queries: write a temp .rb file, pass its path. Bash tool timeout is ~120s — wrap slow queries in WithMaxExecutionTime.timeout_queries(seconds: 30) { ... }. Queries run against the read replica (DATABASE_WORKER_REPLICA1_HOST by default).

Follow-up queries are cheap — start scoped (one record, one field), then drill in as the investigation clarifies. Avoid the urge to return everything in a single large query.

Warm hops: prod_query.sh multiplexes SSH to the bastion (ControlPersist) and reuses the last-good instance IP for 10 minutes (PROD_IP_CACHE_TTL). A second query should skip EC2 discovery. Pin with PROD_INSTANCE_IP to skip it entirely. Tracked in gumroad-private#2197.

Persistent runner: the first replica-default query on a host boots a long-lived rails runner loop (scripts/prod_runner_loop.rb) inside the puma container; later queries are spooled to it and skip Rails boot entirely (~1s warm vs ~14s one-shot). The loop forks per query, keeps stdout/stderr separate, idles out after 30 minutes without work, and is replaced automatically when the local prod_runner_loop.rb changes. Non-default PROD_DB_HOST_VAR sessions (primary writes) always take the one-shot path; PROD_NO_RUNNER_LOOP=1 opts out.

Safety

• Read-only. Never write, update, or delete.
• No Redis KEYS or FLUSH*. prod_query.sh prepends scripts/redis_command_guard.rb, which refuses them: one KEYS over the ~4.6M-key primary blocks it for seconds and 500s the site. Use $redis.scan_each(match:, count: 1000).first(n).
• Always .limit() / .first() / .take() — never unbounded result sets.
• Prefer .pluck(:col, :col) over loading full AR objects.
• Mask PII in output (truncate emails, addresses, payment details).
• .explain before querying large tables without indexed conditions.
• Emit structured output (JSON for complex, .inspect for simple) so results are parseable.

Gumroad-specific gotchas

External IDs, not primary keys

Admin URLs and public IDs use external IDs (ExternalId module — Base64 strings like aBcDeFgHiJkLmNoPqRsTuQ==), not integer PKs.

Purchase.find_by_external_id("aBcDeFgHiJkLmNoPqRsTuQ==")  # correct
Purchase.find("aBcDeFgHiJkLmNoPqRsTuQ==")                  # WRONG — treats it as a PK, silently returns the wrong record

Model naming

User, Balance, Dispute, Follower, CustomDomain behave as names suggest.

Sidekiq queues

critical (12k limit — payouts, webhooks, receipts) → default (300k — general) → long (PDF stamping etc.) → low (expiry). Queue limits at app/controllers/healthcheck_controller.rb:28; worker ordering at docker/web/sidekiq_worker.sh.

DevTools (Gumroad-specific helpers)

See lib/utilities/dev_tools.rb:

• DevTools.reindex_all_for_user(user_id) — reindex ES data
• DevTools.reimport_follower_events_for_user!(user) — reimport follower analytics

For Gumroad-specific scopes and associations (alive, successful, unpaid_balance_cents, payments, products, Flipper checks, Sidekiq introspection), see references/common-queries.md.

Stale bastion host keys look like unhealthy instances

EC2 recycles private IPs, so the bastion's known_hosts accumulates stale keys and refuses the onward hop with REMOTE HOST IDENTIFICATION HAS CHANGED / Offending ECDSA key. From outside this is easy to mistake for a hung instance, and it silently shrinks the usable pool each time instances are replaced.

prod_query.sh catches them without touching the pin: a candidate is recorded as stale-keyed when a changed-key complaint names its address — including on a probe that looked healthy, since the forced command exits 0 after its own hop refused — and the run reports those addresses with the one-hop ssh-keygen -R command to run by hand.

• The pin is never removed automatically. Every textbook trigger for a removal is something a pin exists to defend against: EC2 membership says the address is a pool member, not that the key it serves is the instance's, and a changed-key complaint is an alarm rather than proof of a recycle — deleting the entry and reconnecting pins whatever answered. Clearing is an explicit operator action, printed as a one-hop command.

• Only genuinely outdated keys are reported. A plain timeout, a container that is still starting, or a network blip leaves the recorded key alone — only SSH naming that address counts. The onward hop usually just warns about a changed key and connects anyway (recycled IPs make that the steady state), so a warn-and-proceed failure stays on the patient-retry list too. Only an outright Host key verification failed refusal drops a candidate from the retry.

• A pool rejected entirely for stale keys says so, with that same command, instead of the generic health-probe error.

• LC_PAPER=127.0.0.1 means the bastion itself. The forced command jumps to whatever LC_PAPER names, and omitting the variable — or the -o SendEnv=LC_PAPER that ships it — fails with ssh: Could not resolve hostname; instance IPs run the command on that instance. Anything aimed at the bastion's own files goes through 127.0.0.1.

• The warning text is often noise. A run can print the whole man-in-the-middle banner for a previous hop and still succeed. Judge by exit code and MARK output, not the banner. rc 255 is the transport dropping mid-flight, so the outcome is unknown — do not read it as a failed query. Any other nonzero rc whose only output is the banner means the query failed: look for Query process killed by SIGKILL or the query's own output.