3dgs Articulated Reasoner

3DGS Articulated Reasoner

Reason about articulated objects within 3DGS scenes: extract part structure, infer kinematics, build interactive digital twins.

Capabilities

1. Part Structure Extraction: Given a 3DGS scene containing articulated objects (furniture, vehicles, tools), identify part boundaries and segment Gaussians into movable/ fixed groups.
2. Kinematic Inference: Estimate joint types (revolute, prismatic, fixed) and axes from multi-view observation or user specification. Output URDF-compatible kinematic tree.
3. Part-Aware Compositing: Apply part-aware alpha-compositing that penalizes inter-part penetration at boundaries (inspired by Innovation I-01 in the knowledge base).
4. Digital Twin Generation: Produce an interactable digital twin where Agent commands (open, rotate, slide) drive real-time Gaussian deformation.
5. Simulation-Ready Export: Export articulated 3DGS scenes to simulation frameworks (MuJoCo, Isaac Sim) with preserved visual fidelity.

Instructions

Step 1: Scene Analysis

When the user provides a 3DGS scene or asks about articulated objects:

1. Check references/3dgs-methods-overview.md for relevant methods (ArtiTwinSplat, ArtiSplat, ULF-Loc).
2. Identify the object category and expected articulation pattern from the knowledge base.
3. Determine whether part segmentation is available or needs estimation.

Key references:

• ArtiTwinSplat (arXiv 2026): Articulated digital twin from 3DGS
• ULF-Loc (CVPR 2026): Exposed feature bias at part boundaries
• ArtiSplat: Differentiable physics + rendering for articulated objects

Step 2: Part Segmentation

If the user needs part segmentation:

Input: 3DGS point cloud / rendered views
Method options:
  a) Semantic segmentation (LangSplat/Feature 3DGS) → part labels
  b) Motion-based segmentation (multi-frame observation) → movable vs fixed
  c) User-specified masks (manual annotation on rendered views)
Output: Per-Gaussian part label {0, 1, ..., K}

Validation criteria:

• Part boundaries should align with geometric discontinuities (normal/depth edges)
• Each part should be spatially contiguous (no isolated Gaussians)
• Fixed parts should be the largest spatially connected component

Step 3: Kinematic Estimation

For each part pair with relative motion:

Joint Type Decision Tree:
  IF relative motion is rotational around a fixed axis
    → Revolute joint (hinge)
    → Estimate: axis direction, axis point, rotation limits [θ_min, θ_max]
  IF relative motion is translational along a fixed axis
    → Prismatic joint (slider)
    → Estimate: slide axis, slide limits [d_min, d_max]
  IF no relative motion observed
    → Fixed joint (weld)

Output format: URDF XML string with visual (mesh from TSDF fusion) and collision (simplified convex hull) links.

Step 4: Part-Aware Rendering

When rendering articulated objects for visualization or editing:

# Part-aware alpha-compositing (Innovation I-01)
C(theta) = sum_i T_i * alpha_i * omega_{p(i)}(theta) * c_i

# where omega penalizes penetration:
#   omega = 1.0                    if no penetration
#   omega = exp(-lambda * pen_i)   if penetration detected
# pen_i = SDF_violation at Gaussian center
# lambda = 10.0 (default)

This prevents color bleeding at part boundaries that standard alpha-compositing produces.

Step 5: Agent-Driven Interaction

Support natural-language commands for digital twin interaction:

Deformation pipeline:

1. Load URDF + initial Gaussian positions
2. Apply forward kinematics: new_pos = FK(joint_offsets)
3. Transform Gaussian positions: G_new.mu = R * (G.mu - joint_origin) + joint_origin + t
4. Re-render with part-aware compositing

Step 6: Export & Integration

Export options:

• URDF + Mesh: For MuJoCo / Isaac Sim (convex decomposition via VHACD)
• GLTF with morph targets: For web viewers (Three.js)
• 3DGS-Compatible: Gaussian attributes + joint parametrization for real-time rendering

Reference Data

Articulated Object Methods in Knowledge Base

Common Articulated Object Templates

Penetration Penalty Tuning（无公开基准，须自行实测）

⚠️ 目前没有公开的铰接物体穿透惩罚基准数据集或公认参考值。 λ（穿透惩罚权重）必须在你自己的数据上扫描确定，禁止引用或编造"标准值"。

推荐扫描流程：

1. 以 λ ∈ {0, 1, 5, 10, 20, 50} 做对数网格扫描，每个值训练至收敛。
2. 记录三项指标：穿透体积占比（SDF 判定）、PSNR、关节角误差。
3. 选择穿透显著下降且 PSNR 损失 < 0.2 dB 的最小 λ。
4. 报告时必须注明数据集、迭代数与评估协议（参照 benchmark-data.md 的 [S] 实测标注规范）。