3dgs Training Debugger

Iter     Loss    PSNR    Gaussians   VRAM     Notes
0        0.45    16.2    1(x SfM)    4.2 GB   Init from SfM points
500      0.22    20.1    12,000      5.1 GB   First ADC cycle
1000     0.15    23.5    45,000      7.8 GB   Rapid growth phase
2000     0.09    26.8    180,000     12.4 GB  Growth slowing
5000     0.05    29.2    350,000     14.2 GB  Near convergence
7000     0.04    30.1    380,000     14.5 GB  Fine-tuning
10000    0.03    30.4    390,000     14.6 GB  Final (shutting densif)
15000    0.03    30.5    395,000     14.6 GB  Densif frozen, opacity fine-tune
30000    0.025   30.6    395,000     14.6 GB  Final model

SYMPTOM: Training crashed or NaN
│
├── NaN/Inf in loss?
│   ├── Check gradient norms → extremely large?
│   │   └── Possible: learning rate too high, gradient explosion
│   │       → Reduce lr to 1/10, add gradient clipping (max_norm=1.0)
│   ├── Check after ADC cycle → NaN appears right after densification?
│   │   └── Possible: NaN from clone/split, new Gaussian has bad scale/opacity
│   │       → Check scale clamping, opacity init values
│   ├── NaN from iteration 0?
│   │   └── Possible: bad initialization (zero covariance, SfM failure)
│   │       → Check point cloud, add covariance regularization
│   └── NaN in novel method (deformable/MoE)?
│       └── See Section 9: Novel Method Stability
│
├── CUDA OOM?
│   ├── During training (non-ADC)?
│   │   ├── Gaussian count reasonable but still OOM?
│   │   │   └── Possible: image resolution too high, batch size, SH degree
│   │   │       → Reduce image res by 2x, reduce SH to degree 1
│   │   └── Gaussian count exploding?
│   │       └── Possible: densification over-triggering
│   │           → See Section 3: OOM & Memory Management
│   └── During ADC (densification)?
│       └── Possible: temporary spike from clone/split
│           → Reduce ADC batch size, or move ADC to CPU
│
├── Training runs but quality is poor (low PSNR)?
│   ├── Gaussian count too low?
│   │   └── Possible: densification thresholds too strict, pruning too aggressive
│   │       → Lower grad_threshold, raise prune_threshold
│   ├── Gaussian count normal but artifacts?
│   │   └── See Section 4: Artifact Diagnosis
│   ├── Convergence stalled early?
│   │   └── See Section 6: Convergence Analysis
│   └── Specific views are bad?
│       └── Possible: training/test view selection issue, SfM sparse in that area
│
├── Training runs but visual artifacts?
│   ├── Floaters (small isolated Gaussians)?
│   │   └── See Pattern FP-01 in references
│   ├── Blur / over-smoothing?
│   │   └── See Pattern FP-02
│   ├── Ghosting / duplicate geometry?
│   │   └── See Pattern FP-03
│   ├── Holes / missing regions?
│   │   └── See Pattern FP-04
│   └── Color bleeding / SH artifacts?
│       └── See Pattern FP-05
│
└── Checkpoint resume gives different results?
    └── See Section 8: Checkpoint & Resume

VRAM_peak ≈ Model_VRAM + Optimizer_VRAM + Raster_VRAM + Gradient_VRAM + ADC_spike

Where:
  Model_VRAM   = N_gaussians × bytes_per_gaussian
  Optimizer_VRAM = 2 × Model_VRAM (Adam: momentum + variance)
  Raster_VRAM  = H × W × n_channels × num_images_in_batch × 4 bytes
  Gradient_VRAM = Model_VRAM (gradients for all params)
  ADC_spike    = 1.5 × Model_VRAM (temporary allocation during clone/split)

  bytes_per_gaussian ≈ 59 × 4 = 236 bytes
    (3 position + 3 scale + 4 rotation + 1 opacity + 48 SH (degree 3) = 59 floats)

Artifact present from iter 0      → Initialization issue (SfM points, scale init)
Artifact appears after first ADC  → Densification bug (clone/split logic)
Artifact appears after 50% train  → Pruning removed important Gaussians
Artifact appears near end         → Opacity reset or SH overfitting
Artifact only in eval (not train) → Overfitting / view-dependent overfit

Phase 1: Rapid Growth (iter 0 - 2,000)
  - Loss drops fast, PSNR jumps from ~16 to ~24
  - Gaussian count grows from SfM initial to ~50k-100k
  - Risk: ADC over-triggering → OOM

Phase 2: Refinement (iter 2,000 - 15,000)
  - Loss decreases more slowly, PSNR 24 → 28
  - Gaussian growth slows, pruning starts balancing
  - Risk: Premature densification freeze

Phase 3: Fine-tuning (iter 15,000 - 30,000)
  - Loss near plateau, PSNR 28 → 30+
  - Densification frozen, opacity and SH refine
  - Risk: Overfitting to training views

Phase 4: Final Polish (iter 30,000+)
  - Minimal change, diminishing returns
  - Risk: Continued training may degrade test views

checkpoint = {
    # Model state
    'gaussian_params': {
        '_xyz':      gaussians._xyz.data,        # [N, 3]
        '_features': gaussians._features.data,   # [N, sh_dim]
        '_opacity':  gaussians._opacity.data,    # [N, 1]
        '_scaling':  gaussians._scaling.data,    # [N, 3]
        '_rotation': gaussians._rotation.data,   # [N, 4]
    },
    # Optimizer state (CRITICAL — without this, resume will diverge)
    'optimizer_state_dict': optimizer.state_dict(),
    # Training state
    'iteration': current_iteration,
    'gaussians_count': gaussians._xyz.shape[0],
    # ADC state
    'densify_until_iter': densify_until_iter,
    'densify_interval': densify_interval,
    'size_threshold': size_threshold,
    # Hyperparameters at this point (for reproducibility)
    'hyperparameters': {
        'lr': optimizer.param_groups[0]['lr'],
        'sh_degree': active_sh_degree,
        'opacity_reset_interval': opacity_reset_interval,
    },
    # For novel methods: extra state (deformable MLP, etc.)
    'extra_state': extra_module.state_dict() if extra_module else None,
}