API Mirror

Context

The MirrorService owns the source-agnostic half of a local mirror — the embedded store, the sync-state machine, the runner — so a server supplies only the two parts that are irreducibly per-source: the ingester (a sync generator) and the schema. It targets the embedded-SQLite tier (~10⁴–10⁷ rows). Node/Bun only: bun:sqlite is built-in on Bun, better-sqlite3 is an optional peer dependency on Node; the store is unavailable on Workers (no SQLite, no persistent filesystem).

Import from @cyanheads/mcp-ts-core/mirror.

The shape

import { defineMirror, sqliteMirrorStore } from '@cyanheads/mcp-ts-core/mirror';

const papers = defineMirror({
  name: 'arxiv-papers',
  store: sqliteMirrorStore({
    path: config.mirrorPath,
    table: 'papers',                                 // primary table; FTS index is `papers_fts`
    primaryKey: 'id',
    columns: { id: 'TEXT', title: 'TEXT', authors: 'TEXT', abstract: 'TEXT', updated: 'TEXT' },
    fts: ['title', 'authors', 'abstract'],          // opt-in FTS5 external-content index
    indexes: [{ columns: ['updated'] }],
  }),
  // The ingester — the one part that is always server-specific.
  async *sync({ mode, cursor, checkpoint, signal }) {
    for await (const page of harvestPages({ resumeFrom: cursor, since: checkpoint, signal })) {
      yield {
        records: page.rows,             // objects keyed by declared column
        tombstones: page.deletedIds,    // primary-key values to delete
        cursor: page.token,             // volatile resume position (see below)
        checkpoint: page.maxStamp,      // durable high-water mark (see below)
      };
    }
  },
});

await papers.runSync({ mode: 'init', signal: AbortSignal.timeout(3_600_000) }); // full; resumes on interrupt
await papers.runSync({ mode: 'refresh' });                                       // incremental
const { rows, total } = await papers.query({ match: 'transformers', limit: 10, offset: 0 });
const status = await papers.status();   // { status, ready, checkpoint, total, ... }

cursor vs. checkpoint — the core distinction

Two resume dimensions, deliberately separate. Conflating them silently corrupts resume for token-paged sources.

Why they can't merge: during a from-scratch init the records aren't ordered by the high-water field, so the max-so-far is not a valid resume position — only the cursor is. After a completed init the cursor is meaningless, but the high-water mark is the correct refresh seed. The framework persists both per page and threads the right one back into sync() per mode. The checkpoint must be lexicographically monotonic (ISO 8601 works); the runner advances the stored checkpoint only when a page's value compares greater.

What you own vs. what the framework owns

Querying

query({ match?, filters?, sort?, limit, offset }) covers the common case:

• match — an FTS5 MATCH expression (only when the store declares fts columns). Translate your own query grammar to FTS5 before calling.
• filters — [{ column, op, value }], AND-combined, over declared columns. op ∈ eq|ne|gt|gte|lt|lte|in (in takes an array).
• sort — { column, direction } or 'relevance' (FTS bm25; requires match). Defaults to insertion order.

For access paths the generic query can't express — junction tables for index-backed multi-value filtering, denormalized counters, bespoke bm25 weighting — use the raw handle: const db = await mirror.raw(); then run prepared statements against your own auxiliary tables (declare them via a migration). Add the auxiliary DDL in a migrations step; maintain it from your sync mapping or SQL triggers.

A migration runs identically on first creation and on upgrade: a fresh database runs every migration up to version right after the declarative DDL, an existing one runs only those above its stored version. So up() must tolerate a database that already has the current declarative shape — CREATE TABLE IF NOT EXISTS for auxiliary objects, never an ALTER that assumes an older layout of a declared column.

Readiness — key off the completion marker, not live status

status().ready is true once a full sync has ever completed (completedAt != null), not when status === 'complete'. The dataset stays transactionally queryable during a refresh, so a mirror mid-refresh — or one whose last refresh failed — is still ready and should keep serving. Gate the mirror read path on await mirror.ready(); fall back to the live API only when it is false (cold, never-completed init).