Distilled SDK

Building a distilled SDK

The pipeline

  upstream API description  (a URL, a git repo, a GraphQL endpoint, docs)
    │
    │  stacks/distilled-submodules/spec-repos/<pkg>/fetch-specs.ts   ← you write this
    ▼
  distilled-mirror/spec-mirror-<pkg>          one repo per package, refetched daily
    │                                          layout: .meta/ (machinery) + specs/ (payload)
    │  git submodule
    ▼
  packages/<pkg>/specs/spec-mirror-<pkg>/specs/…
    │
    │  packages/<pkg>/scripts/convert.ts       spec dialect → Smithy
    ▼
  packages/<pkg>/.generated-specs/*.json       Smithy 2.0 models      (committed)
    │
    │  packages/<pkg>/scripts/generate.ts      Smithy → Effect SDK
    ▼
  packages/<pkg>/src/services/*.ts                                   (committed)

Two facts follow from this and shape everything below:

• The generated output is committed. CI never runs generate and never checks out a submodule. A PR is green with no spec present at all — the specs only matter to whoever regenerates.
• The mirror is the spec source, not the upstream. A package never submodules the upstream repository. github/rest-api-description is 6.7 GB checked out for one 13 MB file; the mirror holds the file. See stacks/distilled-submodules/README.md.

Step 1 — identify the upstream shape

There is no generic fetch script. Copy the closest existing one and adapt it; this table is the whole decision.

A YAML spec does not have to be converted in the mirror: spec-repos/coinbase mirrors openapi.yaml verbatim and packages/coinbase/scripts/convert.ts passes parse from the yaml package to runOpenApiConvert. Convert in the mirror only when the upstream is an endpoint rather than a file.

When the user says "like GitHub" they mean a few files out of a big repo — raw download, no clone. "Like cloudflare" means the description has to be scraped out of documentation pages. Copy spec-repos/cloudflare: crawl the sidebar for every page URL, download each page's index.md twin, and fall back to rendering the page's HTML when the twin comes back truncated — that host cuts the markdown off mid-page on its largest schemas. If you are about to scrape docs, first check whether the provider publishes a real machine-readable description somewhere — that is nearly always the better spec.

Rules every fetch-specs.ts follows:

• It runs from .meta/ and writes to ../specs/ — nothing else.
• It validates what it got before writing. A login page, a rate-limit body, or a gutted response is still valid JSON; failing here beats failing three steps later in the generator. The usual check is typeof spec.openapi === "string" && spec.paths !== undefined.
• It writes deterministically — JSON.stringify(spec, null, 2) + "\n" — so a whitespace-only change upstream produces no diff and no daily commit.
• It fetches the subset the generator reads, not the repository.

Step 2 — add the mirror

mkdir stacks/distilled-submodules/spec-repos/<pkg>
# fetch-specs.ts, package.json, readme.md — copy the exemplar from step 1

Add { package: "<pkg>" } to SPEC_REPOS in stacks/distilled-submodules/SpecRepos.ts, in alphabetical position.

You cannot create the mirror repository — it lives in the distilled-mirror org and is created by the distilled-submodules stack, which deploys on merge to main. That is fine; step 3 replaces it.

Step 3 — materialise the mirror locally

pnpm specs:local <pkg>

This copies the same scaffold the stack deploys into packages/<pkg>/specs/.local/ and runs fetch-specs.ts from its .meta directory, so the result is byte-identical to a checkout of the real mirror:

packages/<pkg>/specs/.local/
├── .meta/    fetch-specs.ts, package.json, tsconfig.json
└── specs/    ← what the generator reads

The directory is gitignored. Re-run the command to refetch after editing the fetch script (the pnpm install only happens once).

Step 4 — write the package

Create packages/<pkg>/ from the closest existing package. In convert.ts, declare the production spec path and resolve it:

import { resolveSpecPath } from "@distilled.cloud/core/codegen/spec-path";

const specPath = resolveSpecPath(root, "specs/spec-mirror-<pkg>/specs/openapi.json");

runOpenApiConvert already does this for its specPath entries, so an OpenAPI package gets it for free — just declare the mirror path.

Then register the package: pnpm-workspace.yaml needs nothing (it globs packages/*), but add both tsconfig references to the root tsconfig.json.

Public surface: re-export the generated barrel at the package root (export * from "./services/index.ts") so callers write Pkg.vms.createVm, not Pkg.Services.vms.createVm. Do not add a Services namespace. A single-service package re-exports operations on the root (Pkg.listX) the same way.

Credentials

src/credentials.ts is hand-written (copy packages/s2/src/credentials.ts). Every secret it touches is a Redacted.Redacted<string> from effect/Redacted: API keys, tokens (access, refresh, bearer, session), passwords, client secrets, private keys, and signing or HMAC secrets. Base URLs, account and org IDs, emails, usernames, client IDs and key IDs stay plain strings.

• Inputs take Redacted<string> only. Type a secret parameter of fromApiKey, credentials, fromToken and the like, and any callback that returns one (an OAuth load/refresh), as Redacted.Redacted<string>. Do not type it string, and do not type it string | Redacted<string>. A plain string is how a secret ends up in a log or an error, so the caller wraps it.
• The resolved credentials hold Redacted. The value the Credentials service yields, and any cache of it, keeps each secret redacted.
• Redact environment secrets on read. Use Config.Redacted("<ENV>").
• Unwrap at the point of use. Call Redacted.value only where the header, query string, body or signature is built, and never put a secret in an error message or an error field. Anything minted at runtime (an exchanged OAuth token, a signed JWT) is wrapped as soon as it exists.

This must print nothing:

grep -niE 'readonly \w*(key|token|secret|password)\??: string' \
  packages/<pkg>/src/credentials.ts

Errors and response validation

Every error class in src/errors.ts must be one the protocol can actually raise. A class in the operation error union that nothing constructs tells callers to handle a failure that never happens — that is how ~75 packages ended up declaring a <Pkg>ParseError no code path created.

src/errors.ts declares, and the operation error union (<Pkg>OpError / DefaultErrors) includes:

• Unknown<Pkg>Error — built by the protocol's unknownError fallback.
• <Pkg>ParseError with { body: Schema.Unknown, cause: Schema.Unknown }, .pipe(Category.withParseError) — built by the protocol's parseError (copy packages/s2/src/errors.ts).
• Any status classes the provider needs beyond core's HTTP_STATUS_MAP, each wired into the protocol's statusMap.

The protocol wires every one of them:

• makeRestProtocol requires parseError: parseError: ({ body, cause }) => new <Pkg>ParseError({ body, cause }).
• A hand-written protocol calls validateResponse(outputAst, value, (cause) => new <Pkg>ParseError({ body, cause })) from @distilled.cloud/core/response-validation on every 2xx path that returns the operation output — after wire→TS key mapping, before wrapSensitive. packages/core/src/protocol-rest.ts is the reference.

2xx responses are validated only in strict mode. ResponseValidation (import { ResponseValidation } from "@distilled.cloud/core") is one context reference shared by every SDK: lenient by default, switched with Effect.provide(ResponseValidation.strict) — only ever by layer. A new protocol reads the mode through validateResponse; it never adds its own flag or environment variable.

Lenient mode checks only what the protocol needs in order to transform the body (unwrap an envelope, map keys, wrap sensitive members) and nothing more: a non-JSON body comes back as text, a body missing members comes back as read. Never decode against the output schema outside validateResponse. Strict mode surfaces every spec inaccuracy (an undocumented null, a new enum member) as a <Pkg>ParseError; that is the cost of opting in, and why strict is never the default.

Do not add tests to a generated SDK. Generated code is tested once, through the generator and the protocols in packages/core. packages/core/src/sdks.test.ts runs against every package and checks the glue a new SDK hand-writes: its <Pkg>OpError union reaches <Pkg>ParseError (so catchTag on a strict call typechecks), something outside src/services/ constructs it, and errors.ts exports it. A new package is covered the moment it exists; a package that cannot follow the pattern goes in that file's exemption list with the reason. A package gets a test only for code someone wrote by hand in it — a custom protocol (packages/fly-io/src/protocol.ts, everything in packages/aws), or credentials logic like packages/prisma/test/credentials.test.ts — next to that code. No live tests: calls against real APIs belong to Alchemy's test suite.

Before opening the PR, confirm the parse error and the unknown-error fallback are both constructed outside the generated code — this must print two or more lines:

grep -rnE 'new \w+(ParseError|Unknown\w*Error)\(' packages/<pkg>/src \
  --include='*.ts' --exclude-dir=services

For every other class you add to errors.ts, find where the protocol raises it (statusMap, a code lookup table, or new). A class nothing raises comes out of errors.ts and the error union.

Step 5 — iterate

DISTILLED_SPECS_LOCAL=1 pnpm generate <pkg>

The environment variable re-roots every spec read into .local. It prints a warning to stderr on every run so a local-spec generation is never mistaken for a real one, and it lives in one command's environment — never in a file. Without it the same command reads the submodule, which is what you want once the mirror exists.

Iterate on convert.ts (spec → Smithy) and generate.ts (Smithy → TS) separately: pnpm --filter @distilled.cloud/<pkg> run convert stops after the Smithy models, which is usually where the interesting bugs are.

Patches go in packages/<pkg>/patches/ as RFC-6902 *.json and apply in convert (finalizeConvert at the end of every dialect — OpenAPI, GraphQL, discovery, proto, Cloudflare markdown), so .generated-specs is the patched Smithy model. OpenAPI pointers (/paths, /components) run on the spec before conversion; Smithy pointers (/shapes, /metadata) run on the model after. generate does not patch — it only compiles the committed models.

finalizeConvert is not idempotent — move patches and model transforms only make sense on freshly converted models — so it stamps metadata["distilled.finalized"] and refuses to run over a model that already carries it. To re-apply patches or naming, re-run the package's convert; never post-process a committed .generated-specs file. It also fails if any target in a model points at a shape that does not exist. Stale patch pointers fail the run (they used to warn-and-skip, which is how Fly's whole chain vanished after the spec-mirror prefixed paths with /v1). Pass onStalePatch: "warn" only if you truly want skip.

Operation names are convert policy, not patches, and default to verbNoun (listApps, getApp, createMachine) in the OpenAPI and GraphQL converters and in finalizeConvert (for dialects with no naming step of their own). toVerbNoun only reorders ids it can recognise — go-swagger Apps_list, REST ConfigsList, GraphQL projectCreate — and leaves anything already verb-first or ambiguous (WatchPodList, AppGetOrCreate, accountById) unchanged. Irregulars go in operationNames (lookup by "METHOD path", then operationId) — PUT vs PATCH that share an upstream id need the path key. Cases live in packages/core/src/codegen/rewrite-operation-ids.test.ts (pnpm vitest run); add one before changing the heuristic. Do not RFC-6902-patch /paths/~1foo/get/operationId; those break when upstream adds a prefix. Patch the spec, not the generated TypeScript. Writing and checking a patch is the distilled-sdk-patch skill (.agents/skills/distilled-sdk-patch/SKILL.md).