Foundationdb Advanced Layers

4. The global clock: versions & version-stamps

The sequencer is the only source of "now" that every node agrees on. Use it; never use node-local wall clocks for cross-node decisions.

• tr.GetReadVersionAsync() → the read version: a monotonic, cluster-wide logical clock, identical regardless of which node reads it. Use it for leases / liveness, ordering, "as-of" reasoning.
• tr.CreateVersionStamp() + SetVersionStampedKey/Value → the commit version, assigned atomically at commit. Globally ordered, collision-free → the backbone of queues, logs, and change feeds (§5).
• GetVersionStampAsync() / GetCommittedVersion() → recover the version a transaction committed at.

⚠️ Two clock traps (both real, both bite):

1. Local wall clocks have no shared "now." Comparing a timestamp minted on node A against node B's DateTime.UtcNow is meaningless (skew, drift, NTP steps, VM pauses) — like comparing times across relativistic frames. Cross-node liveness must use the database clock.
2. The version tick-rate is not constant (~1e6/s but it drifts; idle clusters advance slower). So do not convert a version delta into a duration (now - lease > N_versions is unsound). Instead, store a DB-sourced token and test it for change (equality), and measure elapsed time only as the gap between an observer's own consecutive local reads (§5.3).

A shared clock removes skew, but not the fundamental failure-detector impossibility: you still cannot distinguish "slow" from "dead." So liveness is always a policy (a threshold) backed by evict-and-resync, never a proof.

5. Capstone — building a change feed

A change feed lets other nodes observe a stream of changes and maintain an in-memory view. It composes every primitive above. Full compile-checked code: BookStore.ChangeFeed.cs.

5.1 Append-and-signal (in the mutation's own transaction)

Each mutation appends a change under a commit-ordered VersionStamp and bumps a single watched signal key — all in the same transaction as the data write, so the feed can never disagree with the data:

var stamp = tr.CreateUniqueVersionStamp();                       // distinct per change, even several per tx
tr.SetVersionStampedKey(subspace.Key(SUBSPACE_FEED, stamp), FdbValue.ToJson(change));
tr.AtomicIncrement64(subspace.Key(SUBSPACE_SIGNAL));             // wake every subscriber; conflict-free

5.2 Subscribe — cursor streaming with a watch tail

The consumer reads pages after its cursor; when caught up, it watches the signal key (outer token, not tr.Cancellation), awaits outside the transaction, then re-reads. Expose it as IAsyncEnumerable<T> and wrap thinly as a Channel<T> or a callback. The VersionStamp of the last entry is the resume cursor.

5.3 Retention without unbounded growth, and liveness without clock skew

A version-stamped log grows forever, so a GC must trim it. Trim everything consumed by all live subscribers (up to the slowest live cursor); if nobody's live, drop the backlog. "Live" is decided without comparing clocks:

• each subscriber renews a DB-sourced token (its read version) on a local interval;
• an observer reads those tokens on its own local interval and watches for tokens that don't change across several polls (unchanged for N polls ≈ N × the observer's own local delay) — equality-check only, never version→time, never cross-node timestamp comparison;
• the observer's reads are non-snapshot, so a subscriber that renews concurrently conflicts the GC and is spared.

5.4 Fencing — detecting "I fell out of the window" in one round-trip

A subscriber frozen long enough gets evicted and the GC reclaims past its cursor → it missed changes and its view is untrustworthy. It must be told. The efficient signal is a tombstone: when the GC reclaims (·, horizon], it leaves one empty-value entry at the horizon's versionstamp.

• A resumer whose cursor is older than the horizon reads that tombstone first in its normal GetRange — an empty value deserializes to null (a real change is always non-null JSON), so it's detected with no extra read / no serial dependency.
• It throws a typed ChangeFeedOutOfSyncException that propagates through the enumerable / channel / callback; the consumer catches it, reloads current state, and re-subscribes from "now."
• Bonus: the throw aborts the same transaction that would have renewed the stale cursor, so an evicted subscriber never re-registers a misleading cursor.

This is the same contract as Kafka's OffsetOutOfRange / DynamoDB Streams' TrimmedDataAccessException: you can't prevent a too-slow consumer from missing data — you detect it cleanly and force a resync.

6. Distributed-layer review checklist

• No serial read→decide→read chains on the hot path — batched (GetValuesAsync), parallel (Task.WhenAll), or encoded into one read (tombstone-style)?
• Independent reads issued concurrently, never await-ed in a loop?
• Write-hot keys sharded / using atomics; snapshot reads where serialization isn't needed?
• Long scans paged across transactions (5-second window), large values chunked, bulk via Fdb.Bulk.*?
• Cross-node time uses the database clock (read version / versionstamp), never local wall clocks?
• Liveness via token change-detection + local inter-poll elapsed, not version→duration math?
• Unbounded logs/feeds have a retention/GC path, and consumers can detect a gap and resync (fencing)?
• Transaction handlers still idempotent (no external side effects); resolved layer State confined to the transaction?