3dgs MCP Renderer

3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

Architecture

┌─────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ Voice/Text  │────▶│   Agent     │────▶│  MCP Server      │────▶│  3DGS Renderer   │
│ (Whisper/   │     │ (Claude/    │     │  (Node.js/       │     │  (Three.js/      │
│  Prompt)    │     │  TeleClaw)  │     │   Python)        │     │   WebGPU/HiGS/   │
│             │◀────│             │◀────│                  │◀────│   DDF-GS)        │
└─────────────┘     └─────────────┘     └──────────────────┘     └──────────────────┘
                        │                      │                       │
                        │  Tool calls          │  WebSocket/HTTP       │  WebGL/WebGPU/
                        │  (MCP protocol)       │  transport            │  HiGS/DDF-GS

Spec-First Sculpting Pipeline (v0.9.0)

Design inspiration: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) spec-first — define quality criteria and component hierarchy before any rendering; (2) stage-gated sculpting — progressive refinement with acceptance checks at each stage.

Why Spec-First for MCP Rendering?

The original MCP pipeline was reactive: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

• No upfront quality criteria → agent cannot self-assess before rendering
• No stage gates → errors compound across steps (bad camera → bad selection → bad edit)
• No component hierarchy → edits are flat, no part-level control

The fix: Introduce a define_scene_spec tool that runs before any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

The 6-Stage Sculpting Pipeline

┌─────────────────────────────────────────────────────────────┐
│                    SPEC-FIRST SCULPTING                      │
│                                                             │
│  ┌──────────────┐                                           │
│  │ define_scene │  ← Object Spec: component hierarchy,      │
│  │ _spec        │    material system, quality criteria      │
│  └──────┬───────┘                                           │
│         │                                                   │
│         ▼                                                   │
│  Stage 1: blockout    → Bounding boxes, camera framing     │
│         │  gate: bbox coverage ≥ spec.target_coverage?      │
│         ▼                                                   │
│  Stage 2: structural  → Part decomposition, hierarchy      │
│         │  gate: part count & nesting matches spec?         │
│         ▼                                                   │
│  Stage 3: form        → Gaussian density/scale/rotation     │
│         │  gate: PSNR estimate ≥ spec.min_psnr?             │
│         ▼                                                   │
│  Stage 4: material    → PBR/SH assignment per part         │
│         │  gate: material count per part matches spec?      │
│         ▼                                                   │
│  Stage 5: surface     → Normal consistency, thin structures │
│         │  gate: normal consistency score ≥ spec.threshold? │
│         ▼                                                   │
│  Stage 6: lighting    → Environment, shadows, AO           │
│            gate: render quality score ≥ spec.target_score?  │
└─────────────────────────────────────────────────────────────┘

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's blockout → structural → form → material → surface → lighting flow.

Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

1. Execute stage tool (e.g., sculpt_form with parameters)
2. Call render_frame() to get current visual state
3. Call query_scene(query_type="stats") to get quantitative metrics
4. Compare metrics against spec gate criteria
5. If pass → advance to next stage
6. If fail → adjust parameters and retry (max 3 attempts)
7. If 3 failures → report to user with diagnostic info

Voice-Driven Sculpting Example

Loaded on demand — See mcp-tools-spec.md for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

Code-First Rendering Philosophy (v0.9.0)

Design inspiration: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.

Traditional 3DGS Export vs Code-First Export

Hybrid: Procedural Code + 3DGS Splatting

The key insight: not everything needs to be Gaussians. For a desk scene:

• Desk surface → procedural BoxGeometry in code (simple, editable, lightweight)
• Monitor screen texture → procedural MeshStandardMaterial (or 3DGS if view-dependent)
• Complex organic objects → 3DGS splatting data (where procedural code can't compete)

Loaded on demand — See code-first-examples.md for hybrid export code examples.

When to Use Code-First vs Pure Splat

SLAT Connection

The code-first approach connects to SLAT (see ../../references/slat-unified-representation.md): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.

SLAT Latent Editing (v1.0.0)

Theoretical basis: SLAT (Structured Latent Aggregation Transform) — see ../../references/slat-unified-representation.md. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.

Encoding: Scene → Structured Latent

encode_scene_slatent voxelizes the active scene into a regular grid (voxel_size, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a slat_id referencing an in-memory snapshot with an encode_loss (reconstruction RMSE), letting the agent judge fidelity before editing.

Editing in Latent Space

edit_scene_latent applies a LatentEditOp to voxels matched by a LatentSelector (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:

Schema vs core naming: the MCP JSON schema uses snake_case (angle_deg); the internal LatentEditOp uses camelCase (angleDeg). Handlers convert at the boundary. Library/test callers use camelCase directly.

Safety gate: edit_scene_latent computes affected_gaussians; if this exceeds 10% of the scene, the edit is rejected unless confirm=true. This reuses the project-wide 10% safety rule.

Apply to scene: with apply_to_scene=true (default) the edit is re-decoded and broadcast to the renderer via modify_gaussians; with false it only updates the in-memory snapshot, so the agent can preview/cancel before committing.

Decoding: Latent → Scene

Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. delete removes the affected Gaussians entirely.

Voice-Driven SLAT Example

Loaded on demand — See mcp-tools-spec.md for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).

Cross-Scene Latent Transfer & Interpolation (v1.1.0)

v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be transferred to another scene (target), or the two scenes can be interpolated in latent space. Both operations rely on a spatial correspondence built over the voxel grids.

Correspondence: Voxel Grid Matching

Both operations build a voxel grid over the source scene (cell size = match_radius) via buildVoxelGrid, then for each target voxel find the nearest source voxel within match_radius (nearestVoxel, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.

Transferring a Latent Edit

transfer_scene_edit re-applies a LatentEditOp from source to target as a relative change:

• match_radius (default 1.0) bounds the spatial correspondence.
• strength (0–1) controls how strongly the source change is applied; 0 applies nothing, 1 applies fully.
• Safety gate: if the matched fraction exceeds 10% of the target scene, confirm=true is required (same project-wide 10% rule).
• apply_to_scene (default true) re-decodes and broadcasts via modify_gaussians; false only updates the in-memory snapshot for preview.

interpolate_scene_latent

interpolate_scene_latent blends the target scene toward the source in latent space:

• t (0–1): 0 = target unchanged, 1 = fully source. Position, color, and opacity are all linearly interpolated per matched voxel.
• match_radius (default 1.0) governs the correspondence as above.
• Same 10% safety gate and apply semantics as transfer.

Design note: transfer carries relative change (style), while interpolation carries absolute blend (morph). Use transfer to reuse an edit, interpolation to morph one scene into another.

Voice Examples for Cross-Scene Transfer

• "transfer the recolor to the other scene" → transfer_scene_edit (op="recolor")
• "reuse this translate on scene B" → transfer_scene_edit (op="translate", target_slat_id=sceneB)
• "blend scene B toward scene A" → interpolate_scene_latent (t=0.5)
• "morph the table into the desk" → interpolate_scene_latent (t=1.0)

MCP Tools Specification

21 core MCP tools (fully implemented) + 13 experimental tools (schema-only stubs) enable agent-controlled 3DGS rendering, editing, sculpting, latent editing, cross-scene transfer, and export. Full JSON schemas are loaded on demand.

Full tool schemas loaded on demand — See mcp-tools-spec.md for complete JSON schemas, sculpting examples, and reconstruction flows.