Conventions Ingestion

Replay-Safe Ingestion Conventions

Apply to any workflow that turns external or asynchronous input into durable stella state: paginated imports, connector sync, webhooks, file extraction, queue workers, migrations, and repair scans.

Target Property

A retry, duplicate delivery, worker restart, or overlapping run must converge to the same durable state as one successful run. Idempotency is one ingredient; replay safety also requires correct checkpoint ordering, durable retries, and protection from stale work.

Required Design

1. Stable identity. Give every source item a stable, tenant-scoped identity. Enforce it with a database unique constraint whose leading columns preserve the tenant or source boundary. Do not rely on a hash collision check alone.
2. Idempotent persistence. Upsert, claim, or transition by stable identity. Reapplying the same input must not duplicate rows, counters, notifications, or other effects.
3. Explicit outcomes. Distinguish terminal outcomes (applied, unchanged, deliberately rejected) from retryable failures. A skipped item is terminal only when losing it is intentional and auditable.
4. Checkpoint last. Advance a cursor, watermark, or checkpoint only after every earlier item is terminal or has a durable retry record. On an ambiguous failure, hold the old checkpoint and replay.
5. Atomic database batches. For database-only work, persist the items and checkpoint in one short transaction with commitReplaySafeIngestionBatch from apps/api/src/lib/replay-safe-ingestion.ts.
6. External side effects. Object storage, search indexes, email, and remote APIs cannot join the database transaction. Make the database record the source identity/fingerprint and retry state first; use deterministic object keys or provider idempotency keys. Persist the cursor only after all page work is durable.
7. Compare-and-set cursors. Capture the cursor loaded at run start and require it in the checkpoint update. A stale run must return the persisted winner, never overwrite newer progress. Public corpus ingestion uses advanceCorpusIngestionCheckpoint from apps/api/src/lib/corpus-ingestion-checkpoint.ts.
8. Stale-work protection. Mutable inputs need a source version or content fingerprint. A late worker must compare the claimed version before overwriting newer state. AI outputs also include schema, model, prompt, and parser versions in their identity/provenance.
9. Durable execution. Do not rely on detached promises or process memory for required work. Use a durable queue/outbox and deterministic job identity. Add a bounded repair scan when enqueue and commit cannot be atomic.
10. Bounded recovery. Repair scans and list reads use cursor pagination and configured limits. Workers remain stateless and safe under concurrency.
11. Owned schema. The vertical slice owns its source, item, attempt, and failure tables. Shared code provides transaction and identity primitives, not a cross-domain ingestion framework.
12. Bounded external calls. Every remote request has an explicit timeout, bounded retry policy with jitter, provider-aware rate limiting, and a maximum concurrency. Persist retry state; do not hold a database transaction while waiting on the provider.
13. Poison-item isolation. One malformed or permanently rejected item must not stall an entire source forever. Persist the item identity, classified terminal/retryable outcome, sanitized error context, and operator-visible repair path before allowing later progress.
14. Accounted-for source fields. Every field a source states on a page the adapter already fetches is stored, or excluded with the reason. See below.
15. A publisher is not a court. The deciding court comes from the record: the court code in its ECLI first, then the record's own court field, resolved through the shared resolver (apps/api/src/lib/case-law/cz-ecli-courts.ts for Czech sources). Never a per-adapter constant, even where the portal is believed to serve one court's decisions: portals republish other courts, and the court name is what authority weighting is read off. An inventory declaration must name the field the value is read from, and a value produced from a literal satisfies no declaration. no-literal-decision-court rejects a literal court in a case-law adapter.

Checkpoint Boundary

Direct Drizzle writes to syncCursor are banned by no-direct-ingestion-checkpoint-write. Public corpus cursors go through advanceCorpusIngestionCheckpoint; database-only batches keep the write inside the persistCheckpoint callback passed to commitReplaySafeIngestionBatch. The lint rule enforces the visible boundary; it does not prove that preceding external effects are durable.

Source-Surface Census

A page an adapter never fetched is invisible to any inventory of the pages it did. So the declaration comes in two steps, and the first is the surfaces: SourceAdapter.sourceSurfaces is required, and it is total over the addresses the publisher serves for one decision — SOURCE_SURFACES as a kebab-case list, then the map as const satisfies Record<<that union>, SourceSurfaceDisposition> as the surfaces of a literal written as const satisfies SourceSurfaceCensus.

Each surface is one of three things:

• storedSourceSurface(part) — fetched with the decision and kept as that envelope part. The conformance suite reads the part back out of an envelope this repository can produce, so an adapter with no such capture cannot declare one;
• excludedSourceSurface(reason) — the same constructor discipline as the field inventory: a blank reason does not compile, and "not fetched today" is not a reason. A rendering of a payload already kept, a corpus-wide index, a query-scoped export and a page the publisher's robots policy disallows are;
• backlogSurface(adapter, reason) — it belongs in the row and is not there yet. Only an adapter already on source-surface-backlog-baseline.json can be named, each entry is listed there, and source-surface-census.test.ts fails both ways, so the set only shrinks. Recording a surface deletes its line.