Infra Deploy

Deploy

The release process for the beet website. Validate the SAME site across three environments in sequence, each in its own sub-agent, each with an IDENTICAL verification pass:

1. Local proves the build (serve on localhost, verify, stop).
2. Dev proves the cloud path (deploy to dev.beet.org, verify, then tear down, since a standing dev environment is a real monthly cost).
3. Prod publishes the permanent production site (beet.org + www.beet.org).

Run each step as a SEPARATE sub-agent. Do them in series: Dev only after Local is green, Prod only after Dev is green and torn down.

Dev MUST be torn down after verification. Only prod stays up. Dev is a temporary proving ground for the cloud path, not a standing environment: a live dev stack is a real recurring monthly cost. The moment dev verification is green (or fails, or is abandoned), run just beet-destroy and confirm no beet-site--dev--* S3 buckets, no dev Lightsail instance or static IP, and no dev.beet.org DNS record remain. Never leave dev running.

Topology

The deploy block is stage-aware (LightsailBeetSiteBlock, crates/beet_extra/src/infra/templates.rs). One small_3_0 Lightsail box (2 GB, flat monthly price) per stage carries everything: no load balancer, no container registry, no ACM certificate, no VPC of its own.

• http: Caddy on the box terminates TLS for every hostname with an automatic Let's Encrypt cert and reverse-proxies to the beet binary on its app port. Cloudflare fronts the site hostnames PROXIED (an A record at the static IP) and runs Full-strict, so the edge verifies Caddy's cert. The certificate store OUTLIVES the box: a caddy-backup.timer on the box saves /var/lib/caddy to s3://beet-site--<stage>--repo/caddy/ every 15 minutes and cloud-init restores it before Caddy starts, so a rebuilt box serves on the certificate the last one held (good for 90 days, renewed by Caddy a month early) and only the FIRST box of a stage ever issues at boot. The box's IAM key may write that prefix alone, never a release.
• ssh: the beet TUI listens on port 22. Cloudflare does not proxy raw TCP, so ssh rides a DNS-ONLY app.* hostname pointing straight at the static IP. The box's own management sshd moves to 2222 (reachable with the stack's Lightsail key pair). Note this means the Lightsail browser console, which only dials 22, lands on the beet TUI rather than a shell.
• DNS (stage-aware, and this is why --stage matters): dev publishes ONLY dev.beet.org (proxied) + app.dev.beet.org (DNS-only). prod publishes the apex beet.org + www.beet.org (proxied) + app.beet.org (DNS-only). A dev deploy can never touch production apex DNS.

The generic beet binary reads the whole site from ONE store at runtime, the per-stage repo store beet-site--<stage>--repo (the <S3BucketBlock label="repo" {RepoStoreBlock}/> declaration in site/main.bsx), under a per-deploy prefix: <deploy id>/repo/main.bsx, <deploy id>/repo/routes/, <deploy id>/repo/templates/ and <deploy id>/repo/assets/ all live in it, mirrored from the staging dir <RepoStage> assembles in target/infra/beet-site/repo/. The binary sits beside the document as <deploy id>/bin/main-lightsail, the ledger as <deploy id>/ledger.json, and current/ holds the served ledger and the release pointers. The box learns which prefix to read from the release pointer at every start (BEET_REPO in current/main-lightsail.env), so a rollback moves the binary and its document together, and a content-only sync publishes into the prefix the running box already reads. No deployed binary reads beet-site--shared--assets; that bucket is the developers' source of record for site/assets, owned by just site-shared pull|push and untouched by any stage deploy. (The separate beet--shared--assets bucket backs the WORKSPACE ./assets tree the examples, tests and wasm builds read; just beet-shared pull|push. The two trees are not mirrors: blog/ and branding/ belong to the site, and a few workspace-built files (the wasm binary, the geoip database, the robot faces) are borrowed by the <DirCopy/> manifest in site/main.bsx: into the staging dir under the deploy's <RepoStage>, and into site/assets by the local assets verb.) The deployed binary is built with --features aws_sdk,ssh,geoip (see <LightsailBeetSiteBlock features="aws_sdk,ssh,geoip"/> in site/main.bsx), so the served site includes the ssh terminal and country lookups. aws_sdk alone would serve http only.

The site entry declares its own infrastructure. site/main.bsx is a one-shot <CliServer{always:true}> dispatcher whose site is the serve route (the box launches app --repo=s3://.. --server=http,ssh serve, and a developer launches beet --main=site serve --server=http: one process, one argv, the same verb). Its raw analytics, rollup, and archive stores are declared ONCE under the stage <Stack> as writable <S3BucketBlock/> declarations: the deploy provisions them and runtime consumers reach them through the job's own fields, AnalyticsRollupJob{raw: $analytics, rollups: $analytics, archive: $archive}. Each composes <app>--<stage>--<label> from the app_name on the entry's <PackageConfig/>; locally the same declarations attach FsStores under target/stores/<app>--<stage>--<label>. The deploy verbs do not load in the deployed binary at all: a <Fragment bx:cfg="feature:infra && feature:extra"> inside the <Stack> holds them (the stores stay outside it, since an excluded declaration binds to nothing), and it is excluded before the build walk in a binary that links neither crate, leaving a tombstone that reports the condition if anything dispatches into it. A Router is a url space, so the site's urls stay rooted at / and no http/ssh request can reach a deploy route (curl /deploy -> 404).

The box is replaced only when the BOX changes. The instance's user_data renders machine config alone (the blueprint, Caddy, the sshd relocation, the CloudWatch agent, the systemd unit), never a version, so a code-only deploy plans no change to aws_lightsail_instance. The binary instead reaches the box through the repo bucket's stable release pointer, s3://beet-site--<stage>--repo/current/main-lightsail.env, which names this deploy's id, its versioned artifact key and the document prefix it was built against. The unit's ExecStart is a launcher (/usr/local/bin/beet-site-run) that re-reads that pointer at every start, so systemctl restart IS the deploy, and <LightsailRelease/> (after <TofuApply/>) performs it over the management sshd on 2222 and gates on the box actually serving: the RUNNING process reports this deploy's id (read from /proc/<MainPID>/environ), one request through the app's own port on loopback is answered, and NRestarts is unchanged across that whole check. A fetch that fails leaves the installed binary serving rather than killing the unit.

Editing LightsailBlock::build_user_data (bumping CADDY_VERSION, changing the unit, moving the management port) IS a rebuild: terraform cannot update user_data in place, so the instance is destroyed and recreated, and the IAM access key rotates with it (the rotation trigger is keyed on a digest of the rendered script, so the key rotates with every machine-config change; a rebuild from a non-script change, ie a bundle resize, keeps its key). Budget several minutes AFTER the instance reports running for cloud-init to install Caddy, pull the binary from S3 and start the unit; the static IP re-attaches so the address and DNS are stable. Caddy is enabled by cloud-init but STARTED by <LightsailRelease/>, once the apply has re-opened the firewall (started at boot it raced the ports resource, and every ACME attempt against a closed 80 burned Let's Encrypt's per-hostname validation budget). There is still no circuit breaker and no automatic rollback: a binary that fails to boot leaves a broken box. Rolling back is re-pointing the release pointer (beet rollback, which rewrites it) and restarting the unit.

This matters beyond the outage window. small_3_0 is burstable at a 20% baseline per vCPU, and a fresh instance starts at ZERO burst capacity, so a rebuild does the heaviest work of the instance's life while clamped to baseline. That is what produced the cluster of AWS SDK dispatch failure errors within ~75 seconds of unit start on a freshly deployed box. Covered by code_only_deploy_renders_one_box and machine_config_change_rebuilds_and_rotates in crates/beet_infra/src/blocks/lightsail.rs.

beet-deploy exiting 0 does NOT mean the SITE serves. The deploy's last steps are a timed log tail and a cache purge, neither of which gates on readiness, and cloud-init failures do not propagate back to it. <LightsailRelease/> now proves the APP serves: an active unit carrying this deploy's id that answers a request on its own port with no restart underneath, so the old failure where a crash-looping box reported is serving release .. after ~340 restarts is closed, and a deploy exiting 0 while LightsailWatch prints a fatal boot error in that same log should not recur. What the gate deliberately cannot see is everything in FRONT of the app: it probes loopback, so Caddy, DNS and certificate state are all still unproven. ALWAYS curl the site after the deploy returns. Diagnosis path for a bad origin:

• Cloudflare 521 = the edge reached DNS but nothing accepted on the origin's 443. 526 = Caddy answered but its cert is not valid for the Full-strict edge.
• Get a real shell on the management sshd (port 2222, the stack's Lightsail key pair, user ec2-user). Every deploy materialises that key beside the rendered config as target/infra/beet-site/deploy_key.pem (mode 600) for <LightsailRelease/>'s own ssh, so reach for it rather than digging the tofu state's aws_lightsail_key_pair out by hand: ssh -i target/infra/beet-site/deploy_key.pem -p 2222 ec2-user@app.<stage-host>. It is the stage of the LAST deploy, so re-run the deploy's stage before trusting it, and a beet-destroy removes the work directory with it (plan and validate do not write it back), so between dev's teardown and the prod deploy there is no key on disk: a PRE-deploy baseline of the prod box (uptime, NRestarts, the running BEET_DEPLOY_ID) reads private_key off the aws_lightsail_key_pair in the pulled state, written straight into a mode 600 file under .agents/tmp/ and deleted once the deploy has materialised the real one. The state is client-side encrypted, so pull it through beet secrets/exec -- sh -c 'TF_VAR_tf_state_passphrase=$TF_STATE_PASSPHRASE tofu -chdir=target/infra/beet-site state pull > <file>' (the redirect INSIDE the quotes: secrets/exec logs to stdout too): it hands the passphrase to tofu's environment and redacts it from any output, where a secrets/get piped by hand put a prefix of it into a transcript once (2026-09-22). Rotating the passphrase itself is rotate-state per stack (just beet-rotate-state [--stage=prod], and the same verb through site-shared, beet-shared and site-social --stage=prod); the record's own rotation note in secrets.toml has the four commands. Then: systemctl is-active caddy beet-site, ss -lntp, and sudo grep -iE "error|fail" /var/log/cloud-init-output.log.
• beet-site active on its app port while caddy is missing means the TLS terminator failed to install and everything else is fine.

REGRESSION GUARD (fixed): the deploy shipped a binary it never built. A block is declared under its <Stack> rather than as a step in the deploy sequence, so nothing dispatches it and its BuildArtifact's action never ran; <TofuApply/> went on reading that artifact's file off disk, hashing it and uploading it, so every deploy silently shipped whatever an earlier deploy had left in target/ (and on a machine with no prior artifact would fail on a missing file instead). The box then crash-looped on failed to load entry main.bsx: no component, resource or template registered for tag Stack, warning that types newer than the stale binary were unregistered, while beet-deploy reported success. Building now belongs to the step that CONSUMES the artifact: TofuApply calls BuildArtifact::build() immediately before reading each artifact's bytes, so an artifact that is uploaded but never built is unrepresentable. Covered by build_runs_the_declared_process in crates/beet_infra/src/actions/build_artifact.rs. A deploy log with no building: cargo-zigbuild .. line is this bug back.

REGRESSION GUARD (fixed): a rebuild reset the box's firewall and took prod down. aws_lightsail_instance_public_ports names the instance by NAME, which survives a rebuild, so nothing in its own config diffs and the plan leaves it untouched while the replaced instance comes up with Lightsail's DEFAULT rules (22 + 80). State then claims ports AWS has closed. Live, the --store -> --repo argv change replaced the prod box and shut both 443 (522 at every hostname, the site down with a perfectly healthy app behind it) and the management ssh port, so <LightsailRelease/> sat on waiting for ssh on <ip>:2222 (attempt N/60) for 25 minutes; opening the two ports by hand let the release pass INSTANTLY with beet: already serving, which is the tell that the box was never the problem. The ports resource now carries replace_triggered_by on the instance, exactly as the static IP attachment already did. Covered by rebuilding_the_box_reapplies_the_firewall in crates/beet_infra/src/blocks/lightsail.rs. Diagnose with aws lightsail get-instance-port-states --instance-name beet-site--<stage>--main-lightsail-instance: a rebuild deploy whose plan does not list the ports resource is this bug back.

REGRESSION GUARD (fixed): a boot-time Let's Encrypt failure left the box serving TLS the edge rejects, with nothing in the deploy that notices (<LightsailRelease/> probes loopback). Twice live: once the chain download failed after a successful validation, and once (2026-09-11) a rebuilt prod box came up during a Let's Encrypt secondary-validation outage with its first attempts already lost to the firewall race, so beet.org and www.beet.org served 525 for two hours (distinct from 521: something IS listening on 443, its cert is just not one Full-strict accepts) while the certificate the previous box had held was valid the whole time. A rebuild now restores the saved certificate store before Caddy starts, and Caddy starts from the release step rather than racing the firewall. Covered by rebuild_restores_the_certificate_store and release_starts_the_tls_terminator in crates/beet_infra/src/blocks/lightsail.rs. On a 525 check sudo find /var/lib/caddy -name '*.crt' for the MISSING hostname and sudo journalctl -u caddy; if Let's Encrypt itself is failing (During secondary validation, serverInternal) check their status page and community forum before restarting Caddy, since every restart burns validation attempts against a limit of 5 per hostname per hour.

REGRESSION GUARD (fixed): Caddy was installed via its cloudsmith rpm repo, whose setup script writes an amzn/2023 baseurl that Caddy does not publish, with skip_if_unavailable=1. dnf install -y caddy therefore exited No match for argument: caddy, cloud-init logged a failed scripts-user module and carried on, and the box booted serving the app on its own port with NOTHING on 80/443, while beet-deploy reported success and every hostname returned 521. Caddy is now installed from the pinned upstream static release (LightsailBlock::CADDY_VERSION) with its own systemd unit, and the script verifies the binary (caddy version || exit 1) so a failed install stops the boot instead of silently producing a TLS-less box. Covered by installs_caddy_from_static_release_not_rpm in crates/beet_infra/src/blocks/lightsail.rs.

AN INTERRUPTED APPLY LEAVES TWO MESSES, AND BOTH BLOCK THE RETRY. A deploy killed mid-apply (SIGTERM, a lost terminal, a timeout) leaves (1) a stale state lock and (2) resources that exist in AWS but not in state, because tofu creates them before it persists. The retry then fails twice over: first Error acquiring the state lock naming a lock id and the time it was Created, then a wall of EntityAlreadyExists / ResourceAlreadyExistsException. Neither is a code fault and neither self-heals. Recovery, in order:

1. Confirm nothing is actually running (pgrep -af tofu), then tofu -chdir=target/infra/<app> force-unlock -force <LOCK_ID> using the id from the error. The lock is an object in the state bucket (<key>.tflock); deleting it by hand works but skips the id check.
2. Diff tofu -chdir=target/infra/<app> state list against what AWS actually holds, and tofu import <address> <id> each orphan. Prefer import to deleting and recreating whenever the resource is live (a schedule that has fired, a log group holding evidence). The ids are per-type: a log group is its name, an IAM role its role name, an inline role policy <role>:<policy>, a policy attachment <role>/<policy-arn>, a lambda its function name, a scheduler schedule <group>/<name> (ie default/<name>).
3. Re-run the deploy; it should now converge to no-op plus whatever genuinely changed.

CHECK THE REAL EXIT CODE. just beet-deploy ... | sed ...; echo $? reports the exit of sed, so a deploy killed by a signal reads as success. Redirect instead of piping (just beet-deploy --stage=prod > log 2>&1; echo "EXIT=$?") and strip the ANSI afterwards, or set -o pipefail. just prints recipe ... was terminated ... by signal 15 on the line above, which is the real tell.

AN APPLY OUTLIVES MOST TOOL DEADLINES; RUN IT DETACHED. A full deploy takes 10-15 minutes, and a deploy driven from a tool with a kill deadline (an agent harness, a CI timeout) can SIGTERM the mid-flight apply, leaving exactly the two messes above. Run it detached and watch the log instead: setsid nohup sh -c 'just beet-deploy --stage=prod > .agents/tmp/deploy.log 2>&1; echo "EXIT=$?" >> .agents/tmp/deploy.log' &, then tail the log until the EXIT= line appears.

REGRESSION GUARD (fixed): the release step passed a box that never served. <LightsailRelease/> confirmed only that the unit was active and that /proc/<MainPID>/environ carried this deploy's id, but under Restart=always/RestartSec=3 a crash-looping unit ALWAYS has a live MainPID carrying the right id, so a deploy exited 0 while the box 502'd and restarted ~340 times. The converge loop is now the gate: an attempt passes only when the unit is active, the running process reports the expected id, curl -fsS --max-time 5 http://127.0.0.1:<app_port>/ succeeds, AND NRestarts is unchanged across the whole attempt (a crash loop that answers one lucky request between restarts still fails). The probe is loopback deliberately: with a domain the firewall opens 80/443 only, so the app port is reachable from the box alone, which is exactly "the app itself serves". Covered by release_proves_the_running_process and release_probes_the_declared_port in crates/beet_infra/src/blocks/lightsail.rs. A release step reporting success on a box whose NRestarts is nonzero is this bug back.

Do NOT read a single curl: (22) The requested URL returned error: 500 in that step's narration as a failure. The script logs the stderr of EVERY attempt, so one failed probe followed by a pass is a converge loop doing its job, and on a fresh box the first request routinely is that failure: the box's very first / is served while the instance is still at zero burst capacity, and it 500s on an AWS SDK dispatch failure reaching the repo store, the same clamp described under Topology. The tell that it was transient rather than a crash loop is the box log, request #1 -> 500, #2 -> 200 OK, plus NRestarts=0 afterwards. A 500 that repeats across attempts fails the gate on its own and the deploy never reaches this line.

Commands

Every credential the deploy needs (the AWS pair, CLOUDFLARE_API_TOKEN, BEET_SSH_HOST_KEY) is a record of the repo's secrets.toml, loaded into the process environment by the launch before the entry builds; there is no .env. The AWS pair is this repo's own deployer, beet-deployer, scoped by one managed policy per app it declares (beet--deploy, beet-site--deploy, beet-social--deploy) and nothing wider: beet deployer/mint converges the user, its policies and its key, and is the one verb an admin pair has to run (AWS_ACCESS_KEY_ID=.. AWS_SECRET_ACCESS_KEY=.. beet deployer/mint --stage=prod), since a deployer holds no iam: action over itself. A new block whose service the policy does not grant surfaces as an AccessDenied on the next plan and a re-mint fixes it; the account holds no shared deployer pair, so a plan that fails on IAM is this repo's policy and nobody else's. CLOUDFLARE_API_TOKEN is the same shape at the other vendor: one account-owned token per repo, lowered by beet cloudflare/mint (just site-cloudflare-mint) from the cloudflare_ types the stacks render and the Cloudflare actions the routes run, which for this entry is four zone permission groups on beet.org and nothing at the account. Cloudflare has no permissions boundary and no mfa condition, so the group that mints credentials (Account API Tokens Write) is held by a MINT token that lives in the password manager and in no document, and is passed for that one command (CLOUDFLARE_API_TOKEN=.. just site-cloudflare-mint); a 403 from any Cloudflare call names no permission group, so just site-cloudflare-mint --dry-run (no credential needed) is how you see what the declarations ask for. The provider addresses (the region, the Cloudflare account and zone) are not secrets: they are {AwsRegion(..)}, {CloudflareAccount("..")} and {CloudflareZone{..}} spreads on the entry's root <Router> (or a stack), resolved by ancestry and never read from the environment, so a stack with none fails its render naming the spread. Reading it needs this machine's age identity (beet secrets/check says whether it opens, beet secrets/ls lists the records with what each is and where it is re-minted). Always use the beet-* recipes: they build with --features infra,extra, without which the deploy routes load as inert tags and the verb does nothing.

just beet-plan --stage=prod shows the prod plan without applying.

Verification (IDENTICAL for every step)

Each step verifies its environment with the same five checks. Parameters: a BASE_URL (http://localhost:<port> for local, https://dev.beet.org for dev, https://beet.org for prod) and an SSH target (127.0.0.1 + the local ssh port for local; the DNS-only app. subdomain + port 22 for dev/prod, ie app.dev.beet.org / app.beet.org -- the bare dev.beet.org/beet.org are Cloudflare-proxied web-only and do NOT forward ssh). All five must pass.

a. curl (raw http)

GET each key page; assert HTTP 200 and the expected marker:

• / (home) -> 200, renders the landing page
• /docs -> 200
• /docs/design -> 200
• /docs/design/counter -> 200, body contains Counter and You have clicked
• /docs/design/color_schemes -> 200 (the styles page)
• /blog -> 200, and the body carries a /blog/<slug> link per post. The listing is generated by <RouteIndex reverse="true"/> in site/routes/blog/index.bsx rather than maintained by hand, so a scan that found no posts renders an EMPTY page that still answers 200: assert the count, not the status.
• /blog/folk-technology -> 200 (a post reached by its slug, the segment <RoutesDir/> derives from the post's own slug frontmatter rather than its <number>- filename)
• /blog/post-6 -> 301 to /blog/folk-technology. Every post was renamed from post-<n> to a slug, and site/main.bsx keeps the old urls alive with a block of <Redirect path="post-<n>" redirect="<slug>"/>. That block is part of the entry document the box loads out of the repo store, so it is exactly as deployable-and-breakable as a route: sweep all fifteen (for n in $(seq 1 15); do curl -o /dev/null -w '%{http_code} %{redirect_url}\n' "$BASE/blog/post-$n"; done) and assert every one is a 301 to a slug that itself answers 200. A <Redirect/> carries no PageRoute, so it is absent from the sidebar and from RouteIndex, and nothing but this check covers it.

Also fetch /docs/design/counter?color-scheme=light and ?color-scheme=dark and confirm 200 (the scheme is applied server-side for the screenshot check below).

/deploy -> 404. The Router is a url space, so a served request can never reach the sibling deploy commands; a 200 here means the site's routes and the stack's have been merged into one namespace.

/ with Accept: application/json -> 200 with an application/json body (the page serialized by the TemplateRenderer), and an unknown path with the same header -> 404. REGRESSION GUARD (fixed, 2026-09-18 dev): MediaRenderer handed the TemplateRenderer an EMPTY accepts list, which it read as a mismatch against nothing, so every request carrying Accept: application/json was a 500 on every page including the 404 page. Nothing in checks a-d sends that header, so the first evidence was the box log: scanner probes at /graphql and /v2/_catalog logging accept mismatch: requested [], available [Json] beside 500 Internal Server Error while /wp-admin and friends 404'd. Covered by accepts_json_serializes in crates/beet_ui/src/render/media.rs. Scan the box log for 500 after every deploy: a 500 on a path that should 404 is a content negotiation fault, not a scanner.

b. browser verification (navigability, the counter, client errors, mobile layout)

The counter (site/routes/docs/design/counter.bsx) renders a document field as "You have clicked N times.", with a "More" and a "Less" button whose bx:click scripts mutate it. The browser check asserts first paint only: there is no client interactivity tier until the wasm one lands, so a click in the browser does nothing yet. The whole check is one committed test driving the in-house webdriver (chromedriver + a chromium on PATH are the only deps):

BEET_BASE_URL=<BASE_URL> cargo test --test site_browser \
  --features router,json,testing,webdriver -- --include-ignored

tests/site_browser.rs runs everything below and exits non-zero on any client error or overflow. The checks:

Client errors (fail on any). Before navigating, two collectors attach and the run fails if either fires, so a broken client script cannot ship silently -- exactly the miss that let crypto.randomUUID is not a function reach the analytics beacon in production:

• page.console() -- console.error plus uncaught exceptions and failed-request console messages (BiDi log.entryAdded).
• page.responses() -- any 4xx/5xx subresource. A favicon that fails to load raises no console error, so without this a site whose every asset 403s still reports green.

Asset sweep. Load a page that actually carries an image (/blog/folk-technology), collect every /assets/ reference on it (img[src], link[href], script[src], source[src]), fetch each following redirects, and assert 200. This is the store-topology check: the app serves assets from its own bucket, and a private bucket handing out a public url turns every asset into a redirect to a 403. Both this and the response collector were added after exactly that shipped past a green run (S3Store::public_url claimed a virtual-hosted url for a private bucket; it now returns None unless the store is explicitly with_public(true)).

Ignore nothing by default; if a message is genuinely benign, match it exactly and log that it was skipped. CAVEAT: some faults only surface in an insecure context -- crypto.randomUUID/crypto.subtle are gated to secure contexts, and localhost + https:// are both secure, so this check does NOT reproduce that specific bug. The durable fix is keeping secure-context-only APIs out of the client (the beacon now derives its id from crypto.getRandomValues, available on plain http); the collectors still catch the broad class of client JS errors on every env.

Counter + navigability. Headless chromium: goto BASE_URL/docs/design/counter and assert its first paint ("You have clicked 0 times", plus the "More"/"Less" buttons). Then navigate / -> /docs -> /docs/design -> the counter via in-page links at a desktop viewport (the collapsed-nav links are zero-size and unclickable, exactly like a real user; proves the site is navigable, not just direct loads), and load /blog + a post (/blog/bevys-five-and-beets-alive) so the beacon runs on a content page -- the pages the client error was reported on.

Mobile layout (no horizontal overflow). At viewports 375x812 and 320x812, for /, /blog, and /blog/bevys-five-and-beets-alive assert document.documentElement.scrollWidth <= document.documentElement.clientWidth + 1, printing the offending elements on failure so the culprit is obvious. Regression guard: a <pre> code block or a wide embed used to blow <main> past the viewport (<main> is a flex item, fixed with min-width: 0), and the header nav overflowed at 320px (fixed with an @media screen app-bar flex-wrap).

The test asserts ZERO collected client errors across every navigation above; extend tests/site_browser.rs if a deploy needs a check it does not cover, rather than scripting around it.

c. screenshot (styles + color schemes)

Screenshot the home page, the counter page, and /docs/design/color_schemes, each in default, ?color-scheme=light, and ?color-scheme=dark, via the cli: cargo run -p beet-cli -- screenshot '<BASE_URL>/docs/design/counter?color-scheme=light' --output=.agents/tmp/<step>-counter-light.png (an installed beet works too). Save to .agents/tmp/, then Read the PNGs and confirm they render styled (typography, buttons, layout present; light vs dark visibly differ), not an unstyled or broken page. The server-side default scheme is dark, so default and dark are expected to be identical.

d. ssh (the live terminal + multi-tenancy)

The site is also a navigable charcell TUI over ssh (SshTuiServer, multi-tenant). Connect non-interactively over a pty and verify the SAME pages + counter, then prove two clients run at once:

• connect: ssh -tt -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=15 -p <ssh_port> <user>@<host> (local: -p <port> root@127.0.0.1; dev/prod: the app. subdomain app.dev.beet.org / app.beet.org on port 22, any user. The bare dev.beet.org/beet.org records are Cloudflare-proxied and do not forward ssh; only the DNS-only app.* record points straight at the box's static IP. The stable BEET_SSH_HOST_KEY gives a consistent fingerprint, so StrictHostKeyChecking=no is safe.)
• the handshake completing + a rendered frame appearing confirms the ssh server is up.
• multi-tenancy: launch TWO ssh sessions at once (both backgrounded), drive both, and confirm BOTH render the site and respond independently with no crash/hang on either.

Port 2222 on the same host is the box's own management sshd, not the site. Use it (with the stack's Lightsail key pair) to get a real shell for diagnosis: systemctl status beet-site, journalctl -u beet-site, systemctl status caddy.

ssh driver

The driver lives beside this skill, in .agents/skills/infra-deploy/ (it is COMMITTED there on purpose: .agents/tmp is gitignored, and an earlier copy kept there silently vanished between deploys). ssh.sh <HOST> <PORT> runs the whole check (single session + two-client multi-tenancy) and ssh_pty.py <HOST> <PORT> '<SCRIPT>' is the underlying pty driver, python3 stdlib only.

• local: .agents/skills/infra-deploy/ssh.sh 127.0.0.1 <ssh_port> (default 8339; trust the serve output)
• dev/prod: .agents/skills/infra-deploy/ssh.sh app.dev.beet.org 22 / .agents/skills/infra-deploy/ssh.sh app.beet.org 22

The script scales its waits up for any non-localhost host.

Connection (what the driver runs per session): ssh -tt -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=15 -o LogLevel=ERROR -p <port> root@<host> on an 80x24 pty.

Navigation recipe. The TUI is a charcell render of the site: a left-click on a sidebar link navigates (focus also moves with Tab/Shift+Tab and Enter activates, but clicks are deterministic). The driver feeds SGR mouse press+release at 1-indexed cells and reconstructs the final frame through a small VT emulator, then greps it. At 80x24 the site is in its narrow layout: the home/content pages hide the sidebar behind a hamburger menu (the 三/☰ glyph at the top-left), so the sequence opens the menu, navigates, then reopens it once Design has auto-expanded:

1. click the hamburger menu at cell (col 3, row 1): opens the sidebar overlay (Docs/Design/Blog tree).
2. click "Design": loads /docs/design and closes the overlay.
3. click the hamburger again at (col 3, row 1): reopens the overlay, now with the Design subtree expanded (button/color_schemes/counter/...) because the current page is under it.
4. click "counter": loads /docs/design/counter.
5. click the "More" button: increments. On the counter page the sidebar is closed, so the body is full-width and "More" sits at the left, with "Less" beside it. The page title renders fullwidth (Ｃｏｕｎｔｅｒ), so an ASCII grep Counter will NOT match -- grep the normal-width body line You have clicked N times instead (it uniquely identifies the counter page and carries the count).

Everything but the hamburger is clicked BY LABEL, not by cell. The sidebar lists every doc page and every blog post, so adding one page shifts every row beneath it and a pinned m:col,row clicks its neighbour instead -- silently, since the driver has no idea it missed. That is exactly what happened in the Phase 1 shakedown: the Crates docs section landed above Design, so the pinned (6,6) opened /docs/crates/beet_core and all three checks failed on a page they never asked for, while (6,15) in the image check reached the wrong blog post. ssh_pty.py therefore takes t:<text>, which locates text in the CURRENT frame (row-major, cell-aligned so a preceding fullwidth title cannot skew the column) and clicks its first cell, exiting with the frame if it is absent. The hamburger keeps its cell, being the one control no content can move.

As an ssh_pty.py script (w: is a seconds wait, m:col,row a cell click, t: a label click, k: keys): locally w:5;m:3,1;w:2;t:Design;w:3;m:3,1;w:2;t:counter;w:3;t:More;w:2;t:More;w:2 reaches the counter and clicks More twice (expect "clicked 2 times"). Remote hosts double every wait. ssh_pty.py accepts PTY_RULER=1 to print 1-indexed col/row guides, for reading the layout when a label match is ambiguous, and PTY_USER to override the root user.

Two-client recipe: launch two ssh_pty.py runs at once (both backgrounded), drive them to DIFFERENT counts (A clicks More once, B twice), wait, then assert A shows "clicked 1 times" and B "clicked 2 times" and both rendered the counter. Independent counts prove per-session state. ssh.sh does exactly this, and calls out the known crash explicitly if a frame comes back "connection refused/closed/reset".

Image recipe (the third check). The TUI draws real rasters over the cell grid with the kitty graphics protocol, and NONE of the above touches that path: the driver used to advertise a flat xterm-256color with a zero-pixel pty, and KittyGraphicsSupport::from_pty enables graphics only on a kitty/ghostty term name or a non-zero pixel size. A totally broken image path therefore rendered a clean frame and passed. Three things have to line up, and each one silently produces "no image" on its own:

• PTY_TERM=xterm-kitty, else the server never transmits at all.
• Nothing about the window. The check runs at the DEFAULT 80x24: sizing contains a raster within its scroll port and desired_placements places the visible part of a partly clipped one, so the default window draws the post-6 image. It used to ask for PTY_COLS=60 PTY_ROWS=40, which is now gone.
• The image scrolled into view (31 k:down), since it sits well below the fold. The post itself is reached by label (t:Folk Technology, post-6), for the same reason the navigation recipe is.

Assert on PTY_RAW, not the frame: the VT emulator discards APC sequences, which is exactly where the payload lives. ssh.sh greps the raw stream for an ESC _ G a=t transmit and reports the byte count (a healthy run is ~2.7MB), then for an a=p placement carrying the w=/h= source-rect keys, which ride a placement ONLY when the raster is cropped. The second grep is the standing guard on the blank-hole class below: a placement that never crops means either that regression is back or the raster no longer straddles the port, and both want looking at.

FIXED, AND CONFIRMED LIVE (Phase 10): a raster taller than its scroll port used to render as a silent blank hole, no image and no [image]: alt marker. KittyImage::cell_size clamped the width axis to the available columns but nothing bounded rows, so the 1280x960 post-6 photo computed to 30 rows at the default 80x24 window, and desired_placements dropped any rect the screen or clip did not contain whole. Sizing now bounds rows by the nearest scroll port's height and re-derives the columns to hold the aspect (CellBounds, threaded through the measure and layout passes), and a partly clipped placement draws its visible portion through the protocol's source rect instead of disappearing. Covered by tall_raster_fits_the_default_window, raster_bounds_to_its_nested_scroll_port, scrolled_raster_places_a_moving_crop and every_viewport_places_the_raster in crates/beet_ui/src/render/charcell/kitty.rs. Live over ssh at the default 80x24 the photo places as c=64,r=23 (bounded by its port, not the 30 rows aspect alone would give) and crops as it scrolls, the source rect walking h=760 -> h=920 up to the port's bottom edge and then y=40 -> y=120 down the raster.

REGRESSION GUARD (fixed): below the ~90-column breakpoint the sidebar drawer is display: none but position: absolute, and the charcell layout pass placed out-of-flow nodes without consulting display. The invisible drawer therefore kept a full-height rect and won every hit-test in a 20-cell stripe down the left of the screen, so More on the counter page was unclickable over ssh (Tab+Enter still worked). Fixed in crates/beet_ui/src/render/charcell/layout.rs (hidden subtrees are dropped before positioning, and a node that earns no rect this pass is zeroed rather than left holding a stale one). Covered by sidebar_collapses_below_breakpoint there and narrow_viewport_stays_interactive in tests/bsx_site_tui.rs. The default ssh window is 80x24, squarely in the affected range, so a deploy from a checkout without this fix fails check d.

REGRESSION GUARD (fixed): every click inside the open sidebar overlay landed on the item TWO ROWS BELOW at 80x24. The focus ring is a border, so pressing a link moved focus off the hamburger, its ring dropped, the header shrank by two rows and the overlay rose with it before the release arrived; the release was re-hit-tested against the new geometry and reached the wrong link. pointer_input now delivers a release on the pressed cell to the pressed entity (crates/beet_ui/src/render/charcell/hit_test.rs, covered by release_lands_on_the_pressed_entity_after_a_reflow). The in-process twin is cargo test --test bsx_site_tui --features router,style,markdown,fs,tui_server,quickjs,testing (7 tests; quickjs is a required feature, since the counter's bx:click verbs are scripts, and without it cargo skips the target rather than failing it).

KNOWN, AND NOT SEEN SINCE (multi-tenancy crash): an intermittent native crash (SIGSEGV/SIGABRT, no Rust panic even with RUST_BACKTRACE=full) was recorded under two or more CONCURRENT ssh sessions, originally at roughly 20-40% per session pair, and suspected in the russh-on-a-tokio-runtime bridge to the single-threaded bevy world, ie a session despawn racing the per-frame ssh_write drain across sessions. Treat the 20-40% figure as STALE: the crash has not been observed since, across ~17 live session pairs (local, dev and prod, including three concurrent sessions streaming a 2.7MB image) and every run of the harness below.

Phase 6 built the instrument that was missing, and it found a real defect on exactly the suspected path. A connection entity's lifetime belongs to its accept loop, which despawns it the moment the client's channel closes; an entity world scope does not flush, so the surface insert queued by the pty observer was still pending when that despawn ran, and EntityWorldMut::despawn flushes the command queue itself, AFTER removing the entity. Every client that vanished between its pty request and the next flush therefore raised "Entity despawned" through the app's error handler, ie a panic on the server's world thread, and a public port 22 supplies that shape continuously in the form of scanners. The harness hit it ~500 times per 2000-session run. Every command the ssh path aims at a connection is now silenced (try_insert / try_trigger_target / try_remove), pinned by a_session_that_vanishes_before_its_surface_is_not_an_error (crates/beet_router/src/navigate/ssh_tui_server.rs) and try_trigger_target_tolerates_a_despawned_target (crates/beet_core/src/bevy_utils/entity_target_event.rs); both fail without it.

The standing instrument is ssh_stress_multi_tenancy, at the bottom of crates/beet_router/src/navigate/ssh_tui_server.rs: 2000 real russh sessions through a real listener, 24 concurrent, each dropping its socket while its neighbours are mid-frame, one in four vanishing before its own page has finished building. It is #[ignore]d, so the sanctioned test run compiles it and skips it. Run it with

cargo test -p beet_router --features=ssh_stress --lib -- --ignored ssh_stress

18,000 session cycles after the fix (12,000 across six debug runs and 6,000 across three release runs, release being what the box runs) produced zero crashes, zero coredumps and zero raised errors. The roughly 10,000 cycles run before the fix produced the same zero crashes and zero coredumps, and ~500 raised errors per run.

This does NOT prove the historical crash is gone. The harness renders a synthetic page and never exercises the kitty image path's multi-megabyte transmits, and the world-thread blocking fixes that landed earlier may account for the silence on their own. So systemctl show beet-site -p NRestarts stays the free canary on every deploy, and ssh.sh still calls the crash out by name if a frame comes back "connection refused/closed/reset". A sighting now is new information: capture the coredump with coredumpctl BEFORE restarting the unit.

If outbound port 22 is unexpectedly blocked at runtime, fall back to confirming the instance is running with its ports open via aws lightsail get-instance-port-states and record the limitation, but note port 22 was confirmed open from this host at authoring time.