Data Workshop

Data Workshop

A dataset is a source and a chain of cells. Every frame is Parquet; versions are derived, never stored; one gate freezes what consumers use; every edit goes through lifecycle.py, from the chat, REST or MCP.

Model

• Dataset: name, source kind/spec, capability and intent (train | eval | pending, both proposed at landing by alignment.rank / contract.propose_intent and frozen by the first use), capability_rank, active (FK Cell, null = the last cell that ran), state (landing | diagnosing | idle | running | error), chat (the one conversation, a JSON list of turns), agent_id (the Cursor engine's session), agent_messages (the native engine's OpenAI message list, trimmed to a bounded tail) and agent_turn_key (the last Celery task id to run a turn, so an acks_late redelivery is a no-op).
• Cell: one transformation and the frame it left — position (0 = the source), title, script (a Python body over df), note, state (proposed | queued | running | ok | failed), rows, columns, fingerprint, input_fingerprint, intent_report, capability_report, review, quality_report, stats, used_at.
• Frames live only at MEDIA_ROOT/datasets/<id>/cells/<cell>.parquet. Every frame carries source_row: row identity, not data. The runner rebuilds it from the index when a script drops the column; services/datasets/diff.py joins on it for the row-level and value-level diff the grid and the agent show. The grid, the column count and the agent's tool results hide it; the export keeps it.
• The Console also hides _overmind_provenance from table columns, filter choices and column counts. It remains internal row lineage for contamination checks; generation never removes that lineage to change the table presentation.
• Versions (Dataset.versions()): the source is 1.0, each cell after it a minor, a used cell the next major.
• use.check(dataset, intent, cell=) checks the dataset's intent, readable frame and technical format without changing it; validators, readiness tools and estimates call it. use.use / use.freeze set used_at only in the transaction creating the consumer's row, so a refused launch never freezes a version. The FK PROTECTs the cell. A training job pins cell, validation_cell and eval_cell; later consumers read those pinned cells. Used cells and every earlier cell are frozen, together with intent and capability.
• use.check performs the same validation without freezing. Quality findings never block use: missing, failed, unknown, sampled or stale reviews appear as Review recommended in the workshop, and users may continue without approval. Reviews cover task_alignment, input_evidence, answer_support and output_schema. record_quality_review executes a read-only audit script producing source_row and one boolean-or-null column per named check, exactly one result per original row. The server derives results, checked/failed/unknown counts and failing source-row examples; caller-supplied counts and verdicts are not trusted. The script and results are persisted against the frame, intent and capability context. Unexecuted reports cannot yield Quality passed. These are agent-authored audits, not independent proofs of correctness; unknown semantic claims must remain null, and renderer fidelity requires exact comparison with the actual renderer. Declared input/output schemas are checked deterministically with JSON Schema and reported as advisory capability findings; no external schema references are fetched. Dataset summaries expose active-version readiness; MCP cell contracts expose warnings separately from technical fit. Training setup does not repeat workshop review recommendations.
• Dataset names are not unique; every tool result carries the id and the resolver prefers ids. Training warns about train/eval or train/validation overlap by normalized input content, trace/conversation identity, configured groups or synthetic seed lineage. It never silently removes those rows.

Landing and measuring

Console creation requires an explicit purpose: Evaluation, Training, or Train + eval. Capability offers Decide from the rows, None, or a project capability. REST and MCP creation distinguish an omitted capability (infer at landing) from explicit null (leave unbound); dispatch carries that choice to landing for both ordinary creation and both sides of a split. API/CLI callers may still omit intent to request inference.

The Console's New dataset menu chooses Upload file or Data from traces before opening the dialog; pasted rows remain API-only. File selection supports up to 100 removable uploads. Each completed file is counted by POST /api/uploads/{id}/inspect/ with its byte size; source.uploads carries their ids in selection order and land.read_uploads combines the rows before any split. The dialog defaults Train + eval to a 30% evaluation share and previews the backend's half-up row count, with at least one row per dataset. File names, byte sizes and row counts are retained in source_spec.files.

services/datasets/land.py writes cell 0 (files, pasted rows, or traces: one row per trace with the TRACE_MANIFEST columns — identity, runtime, input/output, wire messages/tools, score from the trace's last scored TaskExecution — two queries per chunk of 200 traces, no unit carving; oversized cells are bounded with preview markers), measures it. An llm_calls source lands one row per llm_call span (span_id and origin_trace_id, never trace_id), commits idle, and does not queue a workshop turn. A trace source enters through services/datasets/selection.TraceSource (REST create, the MCP tool and the landing task all parse the same payload): explicit trace_ids or a traces-list selection, unknown filter keys refused, and count() run before the dataset row is created so an empty selection is a 400 / no_traces, never a dataset in error (measure.frame: contract.measure for the intent report, alignment.capability_contract for the row-level one, contract.stats, fingerprint), and proposes capability_rank and the intent. Landing (tasks/datasets.land) runs on its dedicated landing queue, commits both halves atomically as diagnosing (never an intermediate idle), and queues diagnose. Files land unaltered: files.py rejects rows wider than their header, repeated columns and non-UTF-8 input; CSV columns become numeric only when every value spells a number exactly. contract.training_line builds the same {messages, tools?} line for validation and training export. A Landing is the source read once; dispatch.create_split makes a <name> train and a <name> eval dataset with intents fixed and lands both from one read, cut by Landing.split(eval_percent, position) (head | tail | random, at least one row on each side); each source_spec.split names the role and the sibling.

Imports persist a DatasetImport receipt and source manifest before broker publication. The receipt separates queue age from execution leases and fences publication by attempt owner. A running import holds a 5-minute lease that a heartbeat thread renews every 30 seconds, never past the absolute execution limit. An expired lease requeues the import with its exact source while attempts remain; only the last lost attempt blocks it as worker_timeout. Sources for queued, running or blocked imports are retained; duplicate delivery cannot publish a second source. Broker retries and diagnosis handoffs use bounded I/O, stable task IDs and publication ownership. A blocked empty import resumes through the existing run_dataset action (Console Retry import, REST and MCP); the original split and capability choices are preserved. The import reconciler runs every 30 seconds on control. The ordinary busy-state reaper locks stale rows and excludes both targets of any durable import.

Running cells

• notebook/run.py runs the queued cells in position order. A cell whose script and input_fingerprint are unchanged keeps its frame. The first failure stops the run and leaves the rest queued; the last good version stays active.
• notebook/runner.py is the sandbox: an rlimited python3 -I child with pd, np, source and df bound, pandas/numpy file IO disabled, imports audited against notebook/libraries.py (the stdlib, a preloaded tier baked into the worker image, and an installable wheel-only tier that install pulls into MEDIA_ROOT/libraries/<project>/).
• lifecycle.py holds every edit: add_cell, edit_cell (re-queues everything after), remove_cell, accept_proposal, set_active, set_intent, set_capability, delete_dataset.
• run, diagnose and turn run on the interactive queue. Producers persist the task ID and queue timestamp before publication. A matching task claims workshop_started_at once; duplicate or superseded delivery cannot start again. Queue waiting has a separate one-hour deadline. reap_stuck_runs runs every minute and measures the execution limit from the claim, independently of progress updates. A claimed task also touches updated_at every 30 seconds; a started task untouched for 5 minutes is reaped as a stopped worker. An agent retains diagnosing during nested cell execution. Live progress is Redis pubsub (dataset:<id>) replayed over the SSE events/ endpoint.

The agent

The dataset's chat is an independent workshop agent scoped to one dataset's chain; it is separate from the platform's MCP agent surface. notebook/agent.py owns everything the page sees: the Tools class, the one TOOL_SPECS table of tools — status, prepare_examples, query, diff, try_script, inspect, add_cell, edit_cell, remove_cell, set_active, set_intent, set_capability, rename, install, seed_examples, add_synthetic_rows, record_quality_review, check_semantic_quality — the step and event shapes, the persisted turn and billing (charge_llm_usage, service data-workshop, engine and model in the metadata). Every tool result is JSON-safe; every error is {ok: false, error}. status carries each cell's script so edit_cell has something to edit. The system prompt (notebook/prompts.py) inlines the workshop text, the intent playbook, the capability card, the library list and context.workshop_context: shared downstream SFT/model-eval requirements plus whole-frame source and active-version profiles. Profiles group instructions, task labels, input/output shapes and tool schemas; all rows are counted, with 16 retained families and eight clipped examples. Unlisted-family row counts are explicit and require targeted queries. This is structural context, not a semantic audit. MCP inspect_dataset exposes the same preparation_context; the volatile chain arrives through status.

check_semantic_quality executes named semantic questions against actual rows and declared evidence/answer columns, using Jev with generative fallback. Answer support requires separate answer and independent evidence columns. Each call checks at most 200 rows, packed against the transport's UTF-8 state/question budgets; oversized groups split without truncating evidence. Every completed batch checkpoints results and usage before progress is emitted. Repeated calls reread the saved audit and resume the same frame/context/check contract; changed audit checkpoints cannot overwrite concurrent work. Persisted batch IDs make billing reconciliation idempotent on resume. Results are boolean/null, unprocessed and unsupported rows remain unknown, and reports stay advisory. The persisted audit records row identities, decisions, confidence, fallback and usage; status and MCP expose a bounded summary, not the full resumable state. Both quality tools share review.record_quality_results, which checks whole-frame coverage and locks the version/context before merging results. The semantic tool never edits rows, invents labels, or grants approval to a transformation. Deterministic scripts remain the tool for exact rules; generation and semantic-edit approval remain separate.

notebook/engines/ drives the model. engines.select() walks core.model_registry.WORKSHOP_ENGINES — Cursor, then OpenRouter, then the first of OPENAI / ANTHROPIC / GEMINI keys — and both engines take the same tools and emit the same events. Startup requires OPENROUTER_API_KEY, so an engine is always configured. engines/cursor.py runs a resumable Composer session over a workspace notebook/workspace.py writes per turn (AGENTS.md, cells/*.py, frames/<version>.parquet) with the tool table as custom tools. Cursor SDK 1.0.31 or newer restricts the session to the mcp tool group, with shell and subagents disabled; the full workshop context is sent in each request. Batches cannot bypass the tool callbacks through direct service imports. The adapter forwards SDK thinking events, including their reported duration, into the same thinking stream the native engine uses. engines/native.py is a tool-calling loop over core.llms.stream_llm_tools (reasoning on; reasoning_details ride each assistant message within a turn and never persist), capped at MAX_ROUNDS after which one tool-less round writes the report. Context is a character budget (fit): the system prompt is outside it and carries a cache_control breakpoint that the body builder strips for a direct endpoint; oldest tool results compact to a stub first, then whole rounds of earlier turns drop, never the turn in flight. A result over MAX_RESULT_CHARS loses whole items from its largest list and says so (truncated).

add_cell validates before landing and runs directly; try_script is optional. One preview is cached per turn by cell, fingerprint and script, so an identical add_cell consumes it without another sandbox run. Changed inputs or scripts invalidate reuse. Identical pending scripts against the same input, intent and capability context return the existing proposal UUID after verifying its preview. No-op scripts create nothing. Tool receipts omit large preview examples; full proposal previews remain stored for review. Status exposes active_id and flags clipped scripts. The agent's remove_cell accepts only an exact pending-proposal UUID: applied, source and generated versions cannot be erased during cleanup. Proposal file deletion occurs only after the database transaction commits. Approving a later proposal uses a free non-negative position and reverse shifts to preserve both position constraints and all earlier proposals.

prepare_examples applies the shared model-independent transcript conversion in one cell. Initial preparation runs it before the agent; custom cells can also use prepare_examples(df, intent="train" or "eval"). Eval keeps the complete prefix (system, user, prior assistant/tool turns and schemas) in input, and separates the final assistant target into expected_output. Model transcripts are not validated against the application's entry-point schema. Identifier-only eval inputs carry an advisory evidence warning. Collapsing rich context into identifiers requires approval even during initial preparation; it never silently discards evidence.

examples.normalize_record normalises JSON-encoded messages and tools at preparation and consumer boundaries without decoding message content. Tool schema validation runs even on rows with no tool calls; malformed schemas are not dropped. The exact preprocessing cache includes the shared export/validation code fingerprint. Replacing or removing existing system/developer instructions is measured by review.impact and always needs a semantic proposal, including initial preparation; agent edits and automatic reruns cannot bypass this by labelling it mechanical.

Source rows and transformations retain hidden source-content and case identity in _overmind_provenance. It is not a visible data column or a training feature. Existing human_reviewed annotations are inherited by source identity; projection cannot turn unreviewed synthetic examples into human-reviewed data. Contamination checks match packet/onboarding IDs, case IDs and example IDs from input JSON as well as existing lineage, including across SFT/eval projections.

inspect is the measuring tool: it runs a script in the same sandbox with produce_frame=False, requires no df, lands nothing, and returns stdout. It is how the agent computes a threshold before it cuts — query covers whatever SQL can express over every row, inspect covers the rest.

A turn streams chat_step (thinking and tool steps; a thinking step's done part carries the model's reasoning as text when the engine streams it), chat_thinking (reasoning deltas), chat_delta, chat_cell and chat_progress events, each published the moment it happens with a per-dataset seq. The user turn and a running agent entry are saved immediately in Dataset.chat. Tool callbacks serialise per turn and persist the stage, reason, saved counts, steps, narrative text and cell references, so a reload restores progress without SSE replay. Steps and cell references carry UTF-16 text offsets. Both engines use Tools.respond to interleave narration with activity; progress events carry a full saved snapshot. Thinking text is snapshotted at most once per second and capped at 16,000 characters per step. Only provider-exposed text is shown; it is never fabricated. Completion updates that entry with status, ms, engine and model. MCP inspection and get_job(kind=dataset_run) expose the same saved progress. Interactive capacity metrics count durable queued and running datasets, including messages already reserved by workers. scripts/plan_workshop_capacity.py scales on demand per worker using the source-controlled process count, with queue-age and monitor alarms and scale-in suspended.