Clustering System

Clustering System

Hierarchical grid-based clustering for map visualization. Photos are grouped into clusters at 9 zoom levels (0, 2, 4, 6, 8, 10, 12, 14, 16) using a two-tier strategy:

• Global (zoom 0-6): Single query across all verified photos (verified >= 2)
• Per-tile (zoom 8-16): Uses pre-computed tile keys and generated columns for performance

Team clustering is unified into the same clusters table via team_id column (0 = global, N = team-specific).

Key Files

• config/clustering.php — Grid sizes, zoom levels, tile size, TTL, limits
• app/Services/Clustering/ClusteringService.php — Core clustering logic (raw SQL, not Eloquent)
• app/Http/Controllers/Clusters/ClusterController.php — Public API endpoint (GeoJSON + ETag)
• app/Http/Controllers/Teams/TeamsClusterController.php — Team cluster API (GeoJSON + bbox). points() selects summary (not result_string).
• app/Observers/PhotoObserver.php — Dirty tile/team marking + school privacy
• app/Console/Commands/Clusters/UpdateClusters.php — clustering:update (full rebuild)
• app/Console/Commands/Clusters/ProcessDirtyTiles.php — clustering:process-dirty (incremental)
• app/Console/Commands/Clusters/CheckMigrationStatus.php — clustering:check-migration (diagnostic)
• app/Models/Cluster.php — Eloquent model (composite PK, $timestamps = false)
• resources/js/stores/maps/clusters/index.js — Pinia store for cluster data
• resources/js/stores/maps/points/requests.js — Points store: GET_POINTS() with page, year, date, username, signal params
• resources/js/views/Maps/helpers/clustersHelper.js — Frontend cluster rendering + interactions
• resources/js/views/Maps/helpers/mapLifecycleHelper.js — Map init (always adds cluster layer), cleanup, health checks
• resources/js/views/Maps/helpers/pointsHelper.js — Points view, pagination, stats, abort signals
• tests/Feature/Map/Clusters/ClusteringTest.php — Core clustering tests
• tests/Feature/Map/Clusters/ClusteringApiTest.php — API endpoint tests
• tests/Feature/Map/Clusters/TeamClusteringTest.php — Team clustering tests

Artisan Commands

# Full rebuild
clustering:update --populate       # Backfill NULL tile_keys (50k chunks)
clustering:update --all            # Recluster all global + tile zoom levels
clustering:update --team=5         # Cluster a specific team
clustering:update --all-teams      # Cluster all teams with photos
clustering:update --stats          # Show statistics + integrity check
clustering:update --explain        # Show query execution plan (combine with --all)

# Interactive menu (no flags)
clustering:update                  # Shows choice() menu: all, populate, both, team, all-teams, stats

# Incremental (scheduled every 5 minutes)
clustering:process-dirty           # Process dirty tiles + teams
clustering:process-dirty --limit=100 --team-limit=20

# Diagnostic
clustering:check-migration         # Verify columns, indexes, PK, data integrity

Scheduler (Kernel.php): clustering:process-dirty runs every 5 minutes. clustering:update --all --all-teams runs nightly at 00:10.

Invariants

1. Global clustering uses verified >= 2. ADMIN_APPROVED and above. Team clustering uses verified >= 1 (tagged, so school students see their uploads on the team map before teacher approval).
2. PhotoObserver uses two thresholds. Global tile dirty: >= ADMIN_APPROVED. Team dirty: >= VERIFIED.
3. ClusteringService uses raw SQL, not Eloquent. All clustering queries use DB::statement() with INSERT...SELECT. The Cluster model exists but is not used by the pipeline.
4. Team clusters are in the same table as global clusters. team_id = 0 for global, team_id = N for team-specific. All global queries MUST filter WHERE team_id = 0.
5. tile_key must be populated before per-tile clustering works. Always run --populate before --all.
6. Global sentinel tile key is 4294967295 (UINT max). Global zoom clusters use this value. Per-tile clusters use the actual tile key.

Architecture

Photo saved/deleted
  → PhotoObserver marks tile dirty (if verified >= ADMIN_APPROVED)
  → PhotoObserver marks team dirty (if verified >= VERIFIED and has team_id)
    → clustering:process-dirty (scheduler, every 5 min)
      → ClusteringService::clusterTile()    — one tile across zooms 8-16
      → ClusteringService::clusterTeam()    — one team across zooms 0-16

Nightly full rebuild:
  → clustering:update --all --all-teams

Tile Key Computation

Formula with 0.25° tile size (1440 x 720 grid):

latIndex = FLOOR((lat + 90) / 0.25)
lonIndex = FLOOR((lon + 180) / 0.25)
tileKey  = latIndex * 1440 + lonIndex

Database Schema

Clusters table (composite PK)

PRIMARY KEY (team_id, tile_key, zoom, year, cell_x, cell_y)
team_id       UNSIGNED INT      -- 0 = global, N = team-specific
tile_key      UNSIGNED INT      -- 4294967295 = global sentinel
zoom          INT               -- 0-16
year          SMALLINT UNSIGNED -- 0 = all-time
cell_x, cell_y INT              -- Grid cell coordinates
lat, lon      DOUBLE            -- Centroid
point_count   BIGINT UNSIGNED   -- Photos in cluster
grid_size     DECIMAL(6,3)