Alpaca Broker Sse Events

6. Reliability patterns (hard-won)

SSE connections drop — networks, load balancers, deploys, and Alpaca-side resets all happen. A production consumer needs:

1. Heartbeat / silence detection. SSE has no application heartbeat by default. Track lastMessageAt on every frame; if the stream is silent past a threshold (e.g. 5 min), proactively tear down and reconnect — a dead socket often looks "open."
2. Reconnect with exponential backoff + cap. On error/close, reconnect after a delay that doubles up to a ceiling (e.g. start 1s, cap 60s). Reset the delay on a successful connect.
3. A single-reconnect guard. Use a flag so an error storm doesn't spawn many concurrent reconnect attempts racing each other.
4. Always reconnect with since_id = last processed event (see §4).
5. Process idempotently (see §7) — overlap from replay is expected, not exceptional.
6. Don't let a side-effect failure kill the stream. Wrap per-event processing in try/catch; log and continue. One bad event (or a downstream outage) must not stop you consuming the rest.

Note: OpenAPI can't fully model SSE, so generated API clients often hang on these endpoints (waiting for a response that never ends). Use a real streaming HTTP/SSE client, not a codegen'd one.

7. Idempotent processing pipeline

The robust shape for each event:

parse → persist a raw event snapshot (keyed on event_id, skip-if-exists)
      → match the local record by Alpaca ID (account_id / journal_id / order_id / transfer_id)
      → update local state under a row lock / guarded by current status
      → fire side effects (notifications, downstream transfers)
      → advance the stored cursor to this event_id

• Snapshot-first, keyed on event_id. Insert the raw event with an upsert/skip-duplicate on event_id. A duplicate (from replay or an at-least-once redelivery) is then a no-op. This is your dedup boundary.
• Lock the target row (SELECT … FOR UPDATE or equivalent) when mutating a transfer/order so two events for the same record can't race.
• Guard transitions by current status — e.g. only act on a transfer that isn't already in a terminal state, so a late/duplicate "executed" doesn't re-trigger a payout.
• Advance the cursor only after successful processing, so a crash mid-event replays it rather than skipping it (at-least-once, which idempotency makes safe).

8. Per-stream notes

• Account status: drive onboarding UI and "enable trading" off status_to == ACTIVE. Reject sandbox/paper account IDs in live handlers.
• Trade updates: new/accepted/pending_new are pre-fill; update local order state on fill/partial_fill/canceled/rejected. Invalidate any cached portfolio/holdings on fills.
• Journals: remember executed isn't final and correct spawns a new journal ID (see alpaca-broker-journals). Idempotency + ID-keyed snapshots absorb both.
• Funding/transfer: unified stream across 4 entity types; switch on entity_type. Funding-wallet per-transfer status may still need polling (alpaca-broker-funding-transfers).
• NTA: dividends/fees/interest/corporate-actions — persist as activity snapshots; these feed balance/portfolio reconciliation.

9. SSE is necessary but not sufficient

Even a perfect consumer can miss events (extended downtime beyond retention, a bug, an un-handled type). Always pair SSE with a periodic reconciliation/heal pass that re-pulls authoritative state (activities, journals, transfers) from Alpaca and upserts it. SSE is for low latency; reconciliation is for correctness. See alpaca-broker-reconciliation-idempotency.

Related skills: correctness backstop → alpaca-broker-reconciliation-idempotency; dedup/idempotency mechanics → alpaca-broker-reconciliation-idempotency; backoff details → alpaca-broker-rate-limits-resilience; market-data streaming (WS, not SSE) → alpaca-broker-market-data.