Add Connector

Add Connector Skill

Choose the connector runtime first

For Sim Search, use the live provider workflow in the federated Search developer guide. Its browser-safe provider catalog owns provider IDs, API origins, credential aliases, and account modes; its typed runtime registry requires both search and read handlers. ConnectorMeta remains the owner of logos and setup fields. Member mode has no admin resource filters. Service mode requires independent live source verification; resource pickers use shared selectors with canonical manual-input pairs, and plain inputs are fine where no picker applies. Do not implement a Search source by adding a crawler, embeddings, or a scheduled ACL build.

The ingestion instructions below apply to ordinary knowledge-base connectors. Sim Search always uses the live backend, so setting search: true in metadata does not implement federated search; a provider that serves both needs the live handlers and this KB ingestion path.

You are an expert at adding knowledge base connectors to Sim. A connector syncs documents from an external source (Confluence, Google Drive, Notion, etc.) into a knowledge base.

Your Task

When the user asks you to create a connector:

1. Use Context7 or WebFetch to read the service's API documentation
2. Determine the auth mode: OAuth (if Sim already has an OAuth provider for the service) or API key (if the service uses API key / Bearer token auth)
3. Create the connector directory: a client-safe meta.ts (declarative metadata) plus the runtime module that spreads it
4. Register it in BOTH the server registry and the client-safe meta registry

Hard Rule: No Guessed Response Or Document Schemas

If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing.

• Do NOT invent document fields
• Do NOT guess pagination cursors or next-page fields
• Do NOT infer metadata/tag mappings from unrelated endpoints
• Do NOT fabricate ExternalDocument content structure from partial docs

If the source schema is unknown, do one of these instead:

1. Ask the user for sample API responses
2. Ask the user for test credentials so you can verify live payloads
3. Implement only the documented parts of the connector
4. Leave the connector incomplete and explicitly say which fields remain unknown

Directory Structure

Each connector is split into a client-safe metadata file and a server-only runtime file. This mirrors the XBlockMeta / BLOCK_META_REGISTRY split in apps/sim/blocks — client components (the knowledge UI) only need the metadata (icon, name, auth, config fields), so the runtime functions (which pull server-only helpers like input-validation.server → undici → node:net) must stay out of the client bundle.

Create files in apps/sim/connectors/{service}/:

connectors/{service}/
├── index.ts          # Barrel export (re-exports the runtime connector)
├── meta.ts           # ConnectorMeta — client-safe declarative metadata
└── {service}.ts      # ConnectorConfig — spreads the meta + adds runtime functions

• meta.ts exports {service}ConnectorMeta: ConnectorMeta. It imports ONLY the icon from @/components/icons, import type { ConnectorMeta } from '@/connectors/types', and any pure-data constants. It must NEVER import server/runtime code.
• {service}.ts exports {service}Connector: ConnectorConfig. It imports the meta via import { {service}ConnectorMeta } from '@/connectors/{service}/meta', spreads it as the first property, and holds the runtime functions (which may import server-only helpers like @/lib/knowledge/documents/utils).

Authentication

Connectors use a discriminated union for auth config (ConnectorAuthConfig in connectors/types.ts):

See ConnectorAuthConfig in apps/sim/connectors/types.ts: oauth takes provider, requiredScopes, and optional service-account/admin scopes; apiKey takes label, placeholder, and optional.

OAuth mode

For services with existing OAuth providers in apps/sim/lib/oauth/types.ts. The provider must match an OAuthService. The modal shows a credential picker and handles token refresh automatically.

API key mode

For services that use API key / Bearer token auth. The modal shows a password input with the configured label and placeholder. The API key is encrypted at rest using AES-256-GCM and stored in a dedicated encryptedApiKey column on the connector record. The sync engine decrypts it automatically — connectors receive the raw access token in listDocuments, getDocument, and validateConfig.

Connector Structure (meta.ts + runtime)

The declarative metadata lives in meta.ts (ConnectorMeta). The runtime functions live in {service}.ts (ConnectorConfig), which spreads the meta as its first property.

meta.ts — client-safe metadata

import { {Service}Icon } from '@/components/icons'
import type { ConnectorMeta } from '@/connectors/types'

export const {service}ConnectorMeta: ConnectorMeta = {
  id: '{service}',
  name: '{Service}',
  description: 'Sync documents from {Service} into your knowledge base',
  version: '1.0.0',
  icon: {Service}Icon,

  auth: {
    mode: 'oauth',
    provider: '{service}',          // Must match OAuthService in lib/oauth/types.ts
    requiredScopes: ['read:...'],
  },

  configFields: [
    // Rendered dynamically by the add-connector modal UI
    // Supports 'short-input', 'dropdown', and 'selector' types — see ConfigField Types below
  ],

  // Optional: tag definitions are metadata too — declare them here
  // tagDefinitions: [ ... ],
}

Keep meta.ts free of any server/runtime import. Only the icon, the ConnectorMeta type, and pure-data constants belong here.

{service}.ts — runtime (OAuth example)

import { createLogger } from '@sim/logger'
import { fetchWithRetry } from '@/lib/knowledge/documents/secure-fetch.server'
import { {service}ConnectorMeta } from '@/connectors/{service}/meta'
import type { ConnectorConfig, ExternalDocument, ExternalDocumentList } from '@/connectors/types'

const logger = createLogger('{Service}Connector')

export const {service}Connector: ConnectorConfig = {
  ...{service}ConnectorMeta,

  listDocuments: async (accessToken, sourceConfig, cursor) => {
    // Return metadata stubs with contentDeferred: true (if per-doc content fetch needed)
    // Or full documents with content (if list API returns content inline)
    // Return { documents: ExternalDocument[], nextCursor?, hasMore }
  },

  getDocument: async (accessToken, sourceConfig, externalId) => {
    // Fetch full content for a single document
    // Return ExternalDocument with contentDeferred: false, or null
  },

  validateConfig: async (accessToken, sourceConfig) => {
    // Return { valid: true } or { valid: false, error: 'message' }
  },

  // Optional: map source metadata to semantic tag keys (translated to slots by sync engine)
  mapTags: (metadata) => {
    // Return Record<string, unknown> with keys matching tagDefinitions[].id
  },
}

Only map fields in listDocuments, getDocument, validateConfig, and mapTags when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing.

API key connector example

The split is identical — auth lives in meta.ts, runtime functions in {service}.ts.

// meta.ts
export const {service}ConnectorMeta: ConnectorMeta = {
  id: '{service}',
  name: '{Service}',
  description: 'Sync documents from {Service} into your knowledge base',
  version: '1.0.0',
  icon: {Service}Icon,

  auth: {
    mode: 'apiKey',
    label: 'API Key',                       // Shown above the input field
    placeholder: 'Enter your {Service} API key',  // Input placeholder
  },

  configFields: [ /* ... */ ],
}

// {service}.ts
export const {service}Connector: ConnectorConfig = {
  ...{service}ConnectorMeta,
  listDocuments: async (accessToken, sourceConfig, cursor) => { /* ... */ },
  getDocument: async (accessToken, sourceConfig, externalId) => { /* ... */ },
  validateConfig: async (accessToken, sourceConfig) => { /* ... */ },
}

ConfigField Types

The add-connector modal renders these automatically — no custom UI needed.

Three field types are supported: short-input, dropdown, and selector.

// Text input
{
  id: 'domain',
  title: 'Domain',
  type: 'short-input',
  placeholder: 'yoursite.example.com',
  required: true,
}

// Dropdown (static options)
{
  id: 'contentType',
  title: 'Content Type',
  type: 'dropdown',
  required: false,
  options: [
    { label: 'Pages only', id: 'page' },
    { label: 'Blog posts only', id: 'blogpost' },
    { label: 'All content', id: 'all' },
  ],
}

Dynamic Selectors (Canonical Pairs)

Use type: 'selector' for a key declared in the browser-safe selector manifest at apps/sim/lib/selectors/manifest.ts. Remote selectors execute through the authorized selectors.execute server operation and a server attachment; connectors never call providers or resolve credentials in the browser. Apply the add-selector skill when the key does not exist.

Selectors are paired with a manual fallback input using the canonical pair pattern — a selector field (basic mode) and a short-input field (advanced mode) linked by canonicalParamId.

The user sees a toggle button (ArrowLeftRight) to switch between the selector dropdown and manual text input. On submit, the modal resolves each canonical pair to the active mode's value, keyed by canonicalParamId.

Rules

1. Every selector field MUST have a canonical pair — a corresponding short-input (or dropdown) field with the same canonicalParamId and mode: 'advanced'.
2. required must be set identically on both fields in a pair. If the selector is required, the manual input must also be required.
3. canonicalParamId must match the key the connector expects in sourceConfig (e.g. baseId, channel, teamId). The advanced field's id should typically match canonicalParamId (connector config fields differ from block subBlocks here; the block rule that canonicalParamId must not equal the id of a subblock without a canonicalParamId does not apply).
4. dependsOn references the selector field's id, not the canonicalParamId. The modal propagates dependency clearing across canonical siblings automatically — changing either field in a parent pair clears dependent children.

Selector canonical pair example (Airtable base → table cascade)

configFields: [
  // Base: selector (basic) + manual (advanced)
  {
    id: 'baseSelector',
    title: 'Base',
    type: 'selector',
    selectorKey: 'airtable.bases',     // Must exist in lib/selectors/manifest.ts
    canonicalParamId: 'baseId',
    mode: 'basic',
    placeholder: 'Select a base',
    required: true,
  },
  {
    id: 'baseId',
    title: 'Base ID',
    type: 'short-input',
    canonicalParamId: 'baseId',
    mode: 'advanced',
    placeholder: 'e.g. appXXXXXXXXXXXXXX',
    required: true,
  },
  // Table: selector depends on base (basic) + manual (advanced)
  {
    id: 'tableSelector',
    title: 'Table',
    type: 'selector',
    selectorKey: 'airtable.tables',
    canonicalParamId: 'tableIdOrName',
    mode: 'basic',
    dependsOn: ['baseSelector'],       // References the selector field ID
    placeholder: 'Select a table',
    required: true,
  },
  {
    id: 'tableIdOrName',
    title: 'Table Name or ID',
    type: 'short-input',
    canonicalParamId: 'tableIdOrName',
    mode: 'advanced',
    placeholder: 'e.g. Tasks',
    required: true,
  },
  // Non-selector fields stay as-is
  { id: 'maxRecords', title: 'Max Records', type: 'short-input', ... },
]

Selector with domain dependency (Jira/Confluence pattern)

When a selector depends on a plain short-input field (no canonical pair), dependsOn references that field's id directly. Exact references such as {{JIRA_DOMAIN}} remain unresolved in the browser and are resolved only after workspace authorization on the server.

configFields: [
  {
    id: 'domain',
    title: 'Jira Domain',
    type: 'short-input',
    placeholder: 'yoursite.atlassian.net',
    required: true,
  },
  {
    id: 'projectSelector',
    title: 'Project',
    type: 'selector',
    selectorKey: 'jira.projects',
    canonicalParamId: 'projectKey',
    mode: 'basic',
    dependsOn: ['domain'],
    placeholder: 'Select a project',
    required: true,
  },
  {
    id: 'projectKey',
    title: 'Project Key',
    type: 'short-input',
    canonicalParamId: 'projectKey',
    mode: 'advanced',
    placeholder: 'e.g. ENG, PROJ',
    required: true,
  },
]

How dependsOn maps to SelectorContext

The shared connector context builder projects only active dependencies. A canonical dependency uses its active basic or advanced value under canonicalParamId; a non-canonical dependency uses its field id. The resulting key must be a SelectorContextKey in apps/sim/lib/selectors/types.ts and must be explicitly allowed by that selector's manifest entry. The browser sends the connector's workspace scope, not the complete connector configuration.

Available selector keys

Check apps/sim/lib/selectors/manifest.ts for the exhaustive selector keys. Common ones for connectors:

ExternalDocument Shape

Every document returned from listDocuments/getDocument must include:

{
  externalId: string          // Source-specific unique ID
  title: string               // Document title
  content: string             // Extracted plain text (or '' if contentDeferred)
  contentDeferred?: boolean   // true = content will be fetched via getDocument
  mimeType: 'text/plain'     // extracted text; for a format the KB pipeline parses (PDF, Office), set `sourceFile` instead
  contentHash: string         // Change-detection hash (metadata-based when content is deferred)
  sourceUrl?: string          // Link back to original (stored on document record)
  metadata?: Record<string, unknown>  // Source-specific data (fed to mapTags)
}

Content Deferral (Required for file/content-download connectors)

All connectors that require per-document API calls to fetch content MUST use contentDeferred: true. This is the standard pattern — listDocuments returns lightweight metadata stubs, and content is fetched lazily by the sync engine via getDocument only for new/changed documents.

This pattern is critical for reliability: the sync engine processes documents in batches and enqueues each batch for processing immediately. If a sync times out, all previously-batched documents are already queued. Without deferral, content downloads during listing can exhaust the sync task's time budget before any documents are saved.

When to use contentDeferred: true

• The service's list API does NOT return document content (only metadata)
• Content requires a separate download/export API call per document
• Examples: Google Drive, OneDrive, SharePoint, Dropbox, Notion, Confluence, Gmail, Obsidian, Evernote, GitHub

When NOT to use contentDeferred

• The list API already returns the full content inline (e.g., Slack messages, Reddit posts, HubSpot notes)
• No per-document API call is needed to get content

Content Hash Strategy

Deferred-content connectors (contentDeferred: true) must use a metadata-based contentHash derivable from the list response alone, so the sync engine can detect changes without downloading content. Inline-content connectors may hash the content they already hold (computeContentHash).

Good metadata hash sources:

• modifiedTime / lastModifiedDateTime — changes when file is edited
• Git blob SHA — unique per content version
• API-provided content hash (e.g., Dropbox content_hash)
• Version number (e.g., Confluence page version)

Format: {service}:{id}:{changeIndicator}

// Google Drive: modifiedTime changes on edit
contentHash: `gdrive:${file.id}:${file.modifiedTime ?? ''}`

// GitHub: blob SHA is a content-addressable hash
contentHash: `gitsha:${item.sha}`

// Dropbox: API provides content_hash
contentHash: `dropbox:${entry.id}:${entry.content_hash ?? entry.server_modified}`

// Confluence: version number increments on edit
contentHash: `confluence:${page.id}:${page.version.number}`

Critical invariant: The contentHash MUST be identical whether produced by listDocuments (stub) or getDocument (full doc). Both should use the same stub function to guarantee this.

Implementation Pattern

// 1. Create a stub function (sync, no API calls)
function fileToStub(file: ServiceFile): ExternalDocument {
  return {
    externalId: file.id,
    title: file.name || 'Untitled',
    content: '',
    contentDeferred: true,
    mimeType: 'text/plain',
    sourceUrl: `https://service.com/file/${file.id}`,
    contentHash: `service:${file.id}:${file.modifiedTime ?? ''}`,
    metadata: { /* fields needed by mapTags */ },
  }
}

// 2. listDocuments returns stubs (fast, metadata only)
listDocuments: async (accessToken, sourceConfig, cursor) => {
  const response = await fetchWithRetry(listUrl, { ... })
  const files = (await response.json()).files
  const documents = files.map(fileToStub)
  return { documents, nextCursor, hasMore }
}

// 3. getDocument fetches content and returns full doc with SAME contentHash
getDocument: async (accessToken, sourceConfig, externalId) => {
  const metadata = await fetchWithRetry(metadataUrl, { ... })
  const file = await metadata.json()
  if (file.trashed) return null

  try {
    const content = await fetchContent(accessToken, file)
    if (!content.trim()) return null
    const stub = fileToStub(file)
    return { ...stub, content, contentDeferred: false }
  } catch (error) {
    logger.warn(`Failed to fetch content for: ${file.name}`, { error })
    return null
  }
}

Reference Implementations

• Google Drive: connectors/google-drive/google-drive.ts — file download/export with modifiedTime hash
• GitHub: connectors/github/github.ts — git blob SHA hash
• Notion: connectors/notion/notion.ts — blocks API with last_edited_time hash
• Confluence: connectors/confluence/confluence.ts — version number hash

tagDefinitions — Declared Tag Definitions

Declare which tags the connector populates using semantic IDs. Shown in the add-connector modal as opt-out checkboxes. On connector creation, slots are dynamically assigned via getNextAvailableSlot — connectors never hardcode slot names.

tagDefinitions: [
  { id: 'labels', displayName: 'Labels', fieldType: 'text' },
  { id: 'version', displayName: 'Version', fieldType: 'number' },
  { id: 'lastModified', displayName: 'Last Modified', fieldType: 'date' },
],

Each entry has:

• id: Semantic key matching a key returned by mapTags (e.g. 'labels', 'version')
• displayName: Human-readable name shown in the UI (e.g. "Labels", "Last Modified")
• fieldType: 'text' | 'number' | 'date' | 'boolean' — determines which slot pool to draw from

Users can opt out of specific tags in the modal. Disabled IDs are stored in sourceConfig.disabledTagIds. The assigned mapping (semantic id → slot) is stored in sourceConfig.tagSlotMapping.