--- name: geo-assets description: Download and process lunar geo assets (DEMs, stable material albedo, ortho/slope/shade maps, normal maps) with the LunCoSim asset pipeline — Assets.toml entries, ROI cropping, terrain layer wiring in USD, quality presets, bake keys. Use when adding a terrain site to a twin, baking layer maps, or debugging the asset pipeline. --- # Geo assets: download & process lunar terrain for a Twin The application pipeline is composed by `crates/lunco-assets`; the heavy native processors live in `crates/lunco-assets-processing`. Pure Rust — no GDAL. Sources may be **GeoTIFF or PDS3 `.IMG`** (attached or detached `.LBL`; `lunco-assets-processing/src/pds_img.rs`); polar-stereographic products are refused loudly because the crop affine is equirectangular-only. Use the target Twin's own `Assets.toml` as the worked example and inspect its current scene before wiring outputs. Wire baked terrain outputs into a scene through LunCoSim's USD document authoring commands. Open the exact USD source, inspect its layer and target, apply typed `ApplyUsdOp(s)` or the existing terrain-material planner, save, and read back the authored value. Do not patch `.usd*` text directly; if the available command surface cannot author a required standard field, add the smallest typed operation at the owning USD layer first. ## Quick commands (run from this repo's root) ```bash cargo run -p lunco-assets -- list --twin cargo run -p lunco-assets -- download --twin # ALL entries (can be GBs) cargo run -p lunco-assets -- download --twin -a # one entry cargo run -p lunco-assets -- process --twin -a --quality coarse|good ``` `` = a folder holding `Assets.toml` + `twin.toml`. `-a ` = the `[section]` name in its Assets.toml. The same entries appear in-app under Settings ▸ Downloadable data and the Twin inspector once the Twin is open (scanned on open). Asset-consuming domains begin only after the asset owner publishes `TwinAssetMounted`; they must not depend on observer registration order. If a declared dataset is missing, the interactive app asks which entries to download before terrain generation; nothing downloads until the user confirms there or uses a CLI run. Closing a Twin retires its dataset rows under the shared download/process commit barrier and cancels their cooperative tasks before another Twin can reuse the authority. A failed mount or poisoned lifecycle lock is surfaced as an error, never reported as a successful asset postcondition. `--quality coarse` quarters `target_resolution` (floor 64) for a seconds-fast quick-start bake; re-run with `good` (default) for full res. Downloads use the shared `download` section in the user settings file (`lunco-settings::DownloadSettings`): attempts include the first request, delays are exponential and capped, and a failed body read resumes from the received prefix with HTTP `Range` when the source supports it. The CLI and the interactive window therefore have one policy and one cache/path resolver. ## Joining a cropped DEM to the rendered globe The site keeps its own cropped DEM as the only elevation input for its handoff. The DEM asset and retained base grid remain unchanged, and the composed local surface preserves their relief throughout the crop. The crop's border datum sets the visible globe shell radius; celestial and physics state keep the canonical body radius. A worker-prepared exterior collar continues the measured edge profile from the exact crop boundary to the analytic sphere. This collar is visual closure outside the crop; terrain queries and colliders remain on the local surface. Globe tiles cut out the collar footprint with bounded orthographic edge sampling. Do not copy the collar into each tile or make global LOD follow DEM posting density. No body-wide raster asset is required. Exterior smoothing preserves the first native posting and filters only the continuation in its preparation worker, without increasing mesh density. Keep the DEM appearance visibly distinct. See [the geometry contract](../../docs/architecture/60-curvature-elevation-and-gravity.md) for ownership and sampling. Derive the visual collar width from this crop's measured edge relief and one-sided edge slope with a 0.60 relief-grade sizing target. Continue the measured slope over one posting, then fade one nearest-perimeter signal to the sphere. Geometry and appearance share one width sized from maximum edge relief; the native inner posting lattice tapers to an outer boundary with at least 32 segments per side. Keep the DEM and all in-crop heights unchanged. Refine the globe only in the band between the exact crop and the collar's outer cutout; set the local minimum tile size from that handoff footprint, independent of DEM posting density. The collar reads map roles from the cropped DEM prim and appearance from the body's USD-selected `UsdShade` material. A continuation-capable WGSL shader and its USD Shader prim must declare the matching `lunco.lunar-surface-continuation.v1` interface. Rhai admits compatible sources, waits for reflection, and holds simulation on a mismatch. The compositor keeps unadmitted collars hidden; the globe cutout waits for the composed material and visibility. `RunLint` checks the composed declaration against reflected WGSL. Read the body's composed look from `GlobeLod`, with its authored declaration entity as validation provenance. The shader asset path is never selected in Rust. Inspect the actual boundary vertices through bounded `TerrainLodStatus` geometry pages; `mesh_entity` selects a CPU-retained DEM boundary tile, including its morph positions. Use `boundary_only: true` to page only the finite perimeter. When bake sampling math changes, update the persistent visual tile cache revision in the same change. Check both rendered edges against mission queries before accepting a corner join. The active crop supplies its own georeference, posting spacing, border datum, and measured edge profile, so this works with Twin-local crops at different sites and resolutions. A body currently has one built crop handoff; multiple built crops for the same body are a visible input error, not a size-based selection. ### Verify automatic lunar DEM registration For automatic lunar DEM registration, verify both repository fixtures `scenes/tests/lunar_dem_continuation.usda` (omitted coordinates) and `scenes/tests/lunar_dem_georeferenced.usda` (DEM-authored coordinates) in an owned headful production session. The shared continuation scenario waits for material admission and settled DEM streaming. Headless scene tests do not provide the GPU-retained boundary geometry required by this graphics verdict. The scene-site projection and zero defaults are specified in the geometry contract; no tutorial script installs the handoff. ## Where files live (cache resolution) - **Shared cache** — the OS-global cache (`~/.cache/lunco` on Linux, `~/Library/Caches/lunco` on macOS, `%LOCALAPPDATA%\\lunco` on Windows). `LUNCOSIM_CACHE` remains an explicit CI/custom-install override. Every worktree and Twin therefore shares one pool of regenerable data (source libraries, textures, ephemeris, downloaded sources). - **Twin cache** — `/.cache`. A Twin's default-owned downloads land beside the Twin. `twin://` reads resolve `/` first, then `/.cache/`, then the global cache `/`. - **`shared = true`** on an entry sends its write to the global pool instead (`/sources//`) — one download per URL, reused by every twin and worktree. Use it when several Twins intentionally share a multi-GB upstream product; a Twin that must be self-contained should set `shared = false`. - **Raw downloads**: entries without `dest` land in `/sources//` — owner being the twin (`--twin`) or the shared cache (crate manifest). Author `dest` only when a file must sit at a specific path; it is then resolved against that same owner cache. - **Baked outputs** (`output_root = "twin"`): inside the twin at `output`, where the scene's `demSource`/layer attrs expect them. Per-twin, always. ## Git hygiene for downloaded and generated bytes Before downloading a Twin dataset, add and verify ignore rules for the raw cache and generated processing outputs. The canonical policy is: ```gitignore twins/**/.cache/ twins/**/terrain/*/materials/ twins/**/terrain/*/.bakekey ``` Raw downloads belong under the Twin's `.cache` (or the shared global cache when `shared = true`); processed DEMs, maps, and bake stamps are regenerable bytes. Commit `Assets.toml`, processing/reprojection tools, provenance and parameter reports, and USD references. Do not force-add cache or generated terrain bytes. Use `git check-ignore -v ` before staging. A download-only manifest entry is honest when a source projection is unsupported; do not add a misleading `[entry.process]` section just to make a polar product look native. Use an explicit reprojection adapter and record its command and vertical datum, or record the native-tool gap as blocked work. ## Process kinds (in `[key.process]`) | kind | Input | Output | |---|---|---| | `dem` | DTM (GeoTIFF/.IMG) | `/materials/textures/heightmap.tif` — square float32, georef in tags. `output` is a FOLDER; scenes reference it as the search path `demSource = @terrain/@`, found from any scene folder through the Twin root/cache | | `map` | co-registered raster (ortho `.IMG`, `_SHADE`/`_SLOPE`/`_CLRGRAD` `.TIF`) | 8-bit RGB PNG at `output` (a FILE). Gray sources get a 1–99 percentile stretch in linear contrast space, then sRGB encoding for the runtime loader | | `albedo` | grayscale PDS3 `.IMG` or georeferenced TIFF orthophoto | stable linear material-albedo PNG at `output` (a FILE) | | `normalmap` | DTM | DEM-local ENU normal PNG (`RGB = n*0.5+0.5`, decoded by the shared terrain-surface shader kernel) | | `texture` | any image | resized PNG (non-geo default) | | `gltf` | .glb | Draco-normalized .glb; WebP extension conversion pending | The built-in processors are registered through `lunco-assets-processing::process::ProcessorRegistry`. A domain-specific native baker should register a `ProcessorSpec` and publish its sidecars through the shared staging/commit path. Put processor-specific manifest values in `[key.process.parameters]`; do not add a new shared manifest field for every domain. Keep selection, ordering, and onboarding policy in the reusable Rhai `assets` tool library or a Twin-owned script; Rust remains the owner of decoding, heavy math, cancellation, and atomic publication. `DatasetRegistry` rejects process outputs that overlap another process output or any declared download source, including file paths nested under a directory output. The `dem` processor replaces its complete configured folder when it commits. Store independent material maps in sibling paths and point the scene at the DEM folder. Use the existing `assets` Rhai library to process declared sources at runtime: `assets::bake(id)` dispatches `ProcessDataset`, and `assets::bake_scope(scope)` queues each idle processing declaration in a scope. Both read source identity and pipeline settings from `Assets.toml`; the command carries only the dataset id. `ListDatasets` reports completion through each entry's `state`. Calling bake again is safe: the content bake key skips current outputs. Adding another source or changing its output size/ROI therefore needs manifest and Rhai edits only; add a Rust processor only for a new transform. Shared ROI fields: `center_lat`, `center_lon`, `window_m`, `target_resolution = [n, n]`, `pixel_scale_m`, `src_min/max_lat`, `src_min/max_lon`, `frame = "MOON_ME"`, `output_root = "twin"`. For a grayscale orthophoto used as terrain colour, add a separate `albedo` process entry. Its optional native-only parameters are `albedo_base_linear` (neutral material base, default `0.13`), `albedo_detail_strength` (retained local contrast, default `0.35`), and `albedo_illumination_radius_m` (low-frequency field radius, default `40`). The processor writes a stable `albedo.png`; it does not claim to perform full photometric calibration. PDS3 `.IMG` sources supply their projection extent and pixel scale from the attached or detached label. A grayscale TIFF albedo source must author `pixel_scale_m` and all four `src_*` bounds in the process table; the grayscale TIFF decoder does not infer those geographic facts from its tags. Keep `map` for analysis/display outputs. In Rhai, `assembly_builder::lunar_albedo_material_plan(...)` returns standard USD `SetAttribute` operations for the produced albedo/normal assets; submit those through `assembly_edit::batch` or `assembly_edit::propose`. This keeps heavy image math in Rust while making the assembly policy replaceable and extensible. ## Adding a new territory to a twin 1. Find the product: `https://data.lroc.im-ldi.com/lroc/view_rdr/NAC_DTM_`; files under `https://pds.lroc.im-ldi.com/data/LRO-L-LROC-5-RDR-V1.0/LROLRC_2001/DATA/SDP/NAC_DTM//`. 2. For an LROC PDS3 source, read its `.LBL`: `MAP_PROJECTION_TYPE` (EQUIRECTANGULAR → processable; POLARSTEREOGRAPHIC → download-only entry, no `[*.process]`), `MAP_SCALE` → `pixel_scale_m`, and `MIN/MAXIMUM_LATITUDE` + `EASTERNMOST/WESTERNMOST_LONGITUDE` → the four `src_*` fields. **Label longitudes are 0–360 °E — author `center_lon` in the same convention.** Never trust `CENTER_LONGITUDE` (body-frame quirk). For a GeoTIFF, verify its CRS/geotransform is equirectangular, then author `pixel_scale_m` and all four geographic extent fields explicitly; the grayscale decoder does not read those facts from TIFF tags. 3. Pick `center_lat/lon` (the POI), `window_m` (scene size), `target_resolution ≈ window_m / native m-per-px` (square). 4. `sha256 = ""` on first download → the tool prints the hash; paste it in. 5. PDS3 `.IMG` sources declare extent/scale in their label — `src_*` may be omitted (an authored manifest extent wins when all four are set). ## Wiring maps as terrain layers (USD) Maps bind through a **stock UsdShade Material network** — the only authoring path. Bind the Terrain prim to a Material, whose surface output connects to a Shader carrying one `asset inputs:_map` + `float inputs:weight_` per layer. Inspect an existing terrain scene in the target Twin for its exact prim paths before adding a new network: ```usda def Xform "Terrain" ( prepend apiSchemas = ["LunCoTerrainAPI"] ) { string lunco:assetMode = "layered" rel material:binding = # … dem/overzoom/rocks layers … } def Scope "Looks" { def Material "TerrainLook" { token outputs:surface.connect = def Shader "Surface" { uniform token info:implementationSource = "sourceAsset" uniform asset info:wgsl:sourceAsset = @lunco://shaders/terrain_layered.wgsl@ # Optional for streamed CDLOD; omit for a static/root mesh. uniform asset info:wgsl:vertexAsset = @lunco://shaders/terrain_geomorph.wgsl@ asset inputs:albedo_map = @terrain//materials/textures/albedo.png@ float inputs:weight_albedo = 1.0 asset inputs:normal_map = @terrain//materials/textures/normal.png@ float inputs:weight_normal = 0.5 asset inputs:mineral_map = @terrain//materials/textures/slope.png@ float inputs:weight_mineral = 0.0 # raise for a classification drape } } } ``` For authored DEM terrain, `terrain_layered.wgsl` owns the canonical material fragment. `terrain_geomorph.wgsl` is only its optional CDLOD vertex stage; `terrain_shadow.wgsl` is an explicit material for non-authored terrain and is never an automatic recovery choice. Keep shared regolith detail and lunar photometry in the imported `lunco::terrain` and `lunco::lunar` shader modules. See the [terrain rendering decision record](../../docs/architecture/terrain-layered-rendering.md) before adding another terrain shader path. Keep production albedo and normal rasters as authored assets. An illumination- bearing grayscale orthophoto is not intrinsic albedo: declare it as `kind = "albedo"` so native Rust removes its low-frequency acquisition-light field and retains stable local variation around the authored neutral regolith base. Bind the resulting `albedo.png`, which is lit once by the runtime sun and shadows. A calibrated reflectance raster may use `texture` when its colour contract is known. Do not replace an orthophoto with a hillshade, slope, or elevation-colour diagnostic to hide minification aliasing; those remain optional analysis products and their authored role/weight must stay explicit. The render binder still builds missing RGBA8 mip levels once per image asset version, off-thread and with role-aware filtering (linear-light colour, linear scalars, renormalized normals) before enabling trilinear/anisotropic sampling. ### Physics parameters for a DEM generator The `dem` child layer also owns the physics collider-ring lattice. Author these beside `windowM`, `targetRes`, `lodViz`, and `colliderRing` when a Twin needs a non-default contract: For an authored `rocks` layer, `lunco:layer:regionM` is an optional half-extent in metres: omit it or leave it at `0` to cover the whole composed terrain; author a positive value only when a near-field scope is intentional. `lunco:layer:density` is per hectare, and the rendering-quality profile owns the total instance cap. ```usda int lunco:layer:colliderDepth = 8 int lunco:layer:colliderResolution = 49 ``` These values are copied into the typed terrain-generation request and used by native, worker, GUI, and headless physics. They are deliberately independent of `RenderingQualitySettings`, camera-driven visual LOD, and `targetRes`; a graphics preset must never change collider tile count or resolution. Asset paths are **scene-root-relative** and resolve through `twin://`, so they travel with the twin. The generic USD shader projection walks `material:binding` → Material → `outputs:surface.connect` → Shader and publishes one `ShaderLook`; the terrain source reconciler derives the typed roles from that same look. Roles are `albedo`, `mineral`, `surface` (packed R=rough G=AO B=rockDens), and `normal`. The bound Shader also owns the render stages: `info:wgsl:sourceAsset` must expose `@fragment`, and optional `info:wgsl:vertexAsset` must expose `@vertex`. A single WGSL module may provide both. Missing or invalid stages are reported as a structured diagnostic and leave the material unbound. Rust never swaps in a neutral or terrain shader to hide a bad asset; if a Twin wants an explicit non-authored material, author that choice in USD/Rhai so it can be changed without rebuilding the renderer. - Every `inputs:*` is a live-tunable, journaled knob (networked, undoable) and the network is inspectable in usdview/Blender. - **CONNECTED** map inputs are skipped — a connected port is fed by a producer node (doc 18 Tier B), not an authored file. - `mineral` composites **UNLIT after lighting**, so a slope/classification drape stays readable inside shadow — its entire job. - `albedo` is the terrain colour source at its authored `weight_albedo`; at full weight the layered shader disables its procedural dust/mottle colour, while relief normals, roughness, AO, and photometry remain active. - The authored shader source and maps bind on both static and streamed terrain through the same `ShaderLook`; streamed LOD tiles add only their CDLOD geometry inputs, and the derived bake fills slots an authored map left empty. - The runtime derived bake is optional visual refinement after the DEM ground is ready. Its effective resolution is bounded by a static terrain's authored visual target, so a low-resolution static product does not pay for an invisible high-resolution map. The task is cancelled at scene teardown or when its liveness bound expires; the terrain remains usable and the status bus reports the terminal warning. This refinement status is separate from terrain tile streaming, so it cannot hide tile progress or make a presentable ground scene wait for an optional map. - For multi-site scenes, author these inputs **inside a terrain variant** and verify the composed variant through the production scene-test command; do not add a package-specific USD probe for this asset contract. Node-graph authoring is outside this asset pipeline. Read the current multi-domain architecture before introducing a new graph owner. ## Caching & staleness - Downloads skip when the resolved file exists with matching `sha256`. - Bakes stamp a `.bakekey` = `sha256(source bytes ‖ effective config ‖ PIPELINE_VERSION)` beside each output; a matching stamp skips the bake before the expensive decode. Changing the source, ROI, `--quality`, or bumping `PIPELINE_VERSION` (`src/process.rs`) rebakes exactly what changed. Never time-based. - The dataset registry treats that stamp as the completion boundary: a DEM output directory without `.bakekey` is partial and remains downloadable/ processable, even if the directory itself exists. - Processing roots are strict (`cache`, `twin`, or `assets`); an unknown root or missing required owner fails visibly. Processing uses a unique staging output and an atomic commit barrier, so cancellation cannot publish stale terrain. - The terrain derived bake keys through the oracle's canonical surface identity and does not re-hash the full DEM for every request. - A queued DEM build is an indeterminate state until its owner admits a task; do not show a numeric percentage for that scheduler hand-off. Phase changes are discrete status events and active work uses the existing progress entry. - Baked artifacts and the twin cache are gitignored by policy (`terrain/*/materials/`, any `.bakekey` stamp, `.cache/`, and `extern/`) — never commit downloaded or derived payloads. Keep only the `Assets.toml` declaration, source URL/hash, ROI, and processing configuration in Git. ## Gotchas - `*_50CM`/`*_2M` `.IMG` companions are ORTHOPHOTOS (brightness), never elevation — `kind = "albedo"` for terrain colour, `kind = "map"` only for analysis/display, and never `kind = "dem"`. - Confirm the DEM datum and the scene's celestial-body radius before combining terrain elevations with orbital or body-fixed coordinates. Do not encode a product-specific radius correction in the asset pipeline. - Heights are absolute body-datum metres: prims on a surface must be authored at the DEM's own elevation. - The runtime DEM reader requires square rasters; keep the scene's `windowM`/`targetRes` in step with the manifest ROI. - Optional QGIS/GDAL extras (custom-sun hillshade, slope ramps, contours) are external to `lunco-assets`; verify that the Twin supplies and documents its own tooling before invoking it.