Mfs Ingest

MFS — register / update / re-sync data sources

1. What this skill does

Walks the user through getting a data source into MFS so it's searchable. The work splits into:

1. Picking the right connector scheme.
2. Collecting credentials (preferring env:VAR / file:/path indirection over plaintext).
3. Writing a connector TOML.
4. Calling mfs add <uri> --config <toml> and monitoring the returned job.

Each connector has its own field set, credential acquisition story, and gotchas. Per-connector details live in reference/connectors/<scheme>.md — read the matching one before collecting fields for any scheme.

Step 0: Pre-flight (always run first)

mfs --version            # missing? `cargo install mfs-cli` (see install row below)
mfs status               # server reachable? connectors/jobs visible?
mfs config show          # endpoint/profile/client id/server-info debugging
mfs connector list       # what's already configured?

Branch on the result:

Step 1: Classify intent (the central decision)

Read the user's most recent message. Pick exactly one row:

Mid-flow redirect

If at any point the user changes intent ("wait, just list what I have" / "actually let me just re-sync the existing one"), abandon the current § and jump to the new one. Don't insist on finishing the original branch.

§A. Zero-friction add

User knows what to add and has credentials handy. Aim for: ≤3 questions to the user, then write toml + run mfs add.

1. Parse the URI from the user's message. Shape: <scheme>://<alias>. Scheme is required and is the connector type (postgres, slack, …). Alias is the human-readable instance ID — gets used as the toml filename and the connector's row in metadata.

   • If only <scheme> was given (no alias), ASK: "What should I call this instance? (free-form; appears as the URI host part, e.g. postgres://**prod-db**)"
   • file takes a bare path, not an alias. The target is a local path: mfs add /abs/path (the URI is derived as file://local/abs/path). The path is client-side: on the same host the server reads it directly; on a different host the CLI bundles and uploads the tree (--upload / --no-upload to force).

2. Read the matching reference/connectors/<scheme>.md for the required field set, and reference/credentials.md for how credentials work. For each credential field, put a reference in the toml — env:VAR_NAME or file:/abs/path — which the server resolves against its own environment / filesystem at ingest time. Make sure that value is present where the server runs (client and server share a machine on a loopback endpoint; otherwise it lives on the server — ask the user if unsure).

3. Write the toml to a temp path:

   # mfs-server connector config — <scheme>
   # URI: <uri>
   <field1> = "<value or env:VAR>"
   <field2> = "<value>"
   ...

   Use a path like /tmp/mfs-<alias>.toml so it doesn't pollute the user's cwd.

4. Run mfs add in estimate-confirm mode for external sources where cost matters (databases >100k rows, GitHub repos with many issues, large Slack workspaces, full website crawls):

   mfs add <uri> --config /tmp/mfs-<alias>.toml

   For non-local targets, the current CLI automatically calls /v1/connectors/estimate and prompts Continue? [y/N] unless --yes is set. There is no standalone --estimate flag. Show the estimate to the user and only answer yes when the user has approved.

   For small / unambiguous sources (single repo of docs, one CRM with <10k records, a defined Slack channel), the same command is still the normal add path. Use --yes only when the user has already accepted skipping the estimate confirmation.

   Whole-account enumerators (gdrive = the entire Drive; feishu user-mode docs = the entire My Space): if the estimate is large, don't just confirm a full index — first propose narrowing by time. Re-estimate with a recent start date (POST /v1/connectors/estimate with a since field) to show the smaller count, then add with that bound:

   mfs add <uri> --config /tmp/mfs-<alias>.toml --since <date>

   --since indexes only objects modified on/after <date>; older ones are left untouched and never deleted, and can be pulled in later by lowering --since.

5. Capture the queued job id. If the step 4 command was approved at the prompt, it already queued the job. For local targets, or when the user has explicitly approved skipping the estimate confirmation, run:

   mfs add <uri> --config /tmp/mfs-<alias>.toml

   Capture the returned job_id. mfs add always returns after queueing; use mfs job show or mfs job list to watch terminal state.

6. Follow the job until terminal state:

   mfs job show <job_id>
   # or polled (no jq needed — grep the JSON status field):
   while ! mfs job show <job_id> | grep -qE '"status": *"(succeeded|failed|cancelled)"'; do
     sleep 5
   done
   mfs job show <job_id>

7. Confirm result — report what's searchable, not just what was registered:

   • succeeded + succeeded_objects > 0 → run mfs connector inspect <uri> and report both numbers: object_count (files registered) and objects.indexed / chunk_count (files actually embedded and semantically searchable). They often differ — only documents, code, and (with a vision model on) images get embedded; data / config files (.json .csv .yaml .log …) are listed and greppable but not vector-searchable. Don't claim "all N indexed" when only some are. Then give one example: "Try: mfs search '<sample query>' <uri>".
   • succeeded + succeeded_objects == 0 → check mfs ls <uri> — either source genuinely empty, or wrong text_fields/scope. Read reference/troubleshooting.md.
   • failed → read the job's error field, match against reference/troubleshooting.md, propose a fix and ask user.

§B. Greenfield walk-through

User vague about what to add. Hand-hold through scheme picking, then delegate to §A's steps 2-7 with the chosen scheme.

1. Pre-flight (Step 0 already covered this).

2. Ask: which kind of source? Group the 20 schemes by shape so the choice is tractable:

   Pick the source TYPE:
     1. Database tables       (postgres, mysql, snowflake, bigquery)
     2. Document store        (mongo)
     3. Code repository       (github)
     4. Issue tracker / wiki  (jira, linear, notion)
     5. CRM                   (hubspot)
     6. Support / help desk   (zendesk)
     7. Chat / messaging      (slack, discord, gmail, feishu)
     8. Cloud storage / files (s3, gdrive, file, web)
     9. Other (specify)

   Once user picks a group, narrow to the specific scheme (e.g. "Database tables → postgres / mysql / snowflake / bigquery — which?").

3. Ask for an instance alias (host part of the URI; e.g. "prod-db", "support-workspace", "main-repo").

4. Read reference/connectors/<scheme>.md — its top section "How to obtain credentials" guides the user through fetching the token/DSN/key from the source's own console. Walk them through one step at a time, ask after each step ("Got the token? Paste it as env:VAR_NAME if it's already exported, or paste the value here").

5. Continue with §A from step 2 (collect fields → write toml → estimate-confirm/add → follow job → confirm).

§C. Force re-ingest

User wants to re-sync an existing connector — typically because the source changed (new tickets, new PRs, new files) and the user doesn't want to wait for the next scheduled sync.

1. Confirm the URI matches a registered connector:

   mfs connector list | grep <alias-or-scheme>

   If not found, redirect to §B.

2. Confirm with the user when it's a force-full re-index (re-embeds everything, costs tokens):

   "Re-syncing <uri>. Pick one: • no flag pull changed data using the connector's normal sync path • --since limit to changes since a date — only on connectors that support it (currently gdrive, feishu); others return an error • --full re-embed everything from scratch (re-bills embedding API; only do this if you've changed text_fields, the embedding model, or chunking config)"

3. Run:

   mfs add <uri>                  # incremental: re-uses existing toml + caches
   mfs add <uri> --full           # full re-embed
   mfs add <uri> --since <date>   # only new content since date

4. Follow + confirm as in §A step 6-7.