--- name: add-discovery-type description: >- Stand up a brand-new RunWhen Local discovery platform / indexer from scratch (a new cloud or resource source) following the native SDK pattern used by azureapi and gcpapi. Use when asked to add a new platform, new cloud provider, or new top-level discovery source. Covers the parity registry, indexer modules, platform handler, tag/hierarchy templates, pipeline wiring, tests, and docs. --- # Add a new discovery type Use this to add a **new platform** (a new cloud provider or discovery source). To add a single resource type to an existing indexer, use the `extend-discovery-type` skill instead. `gcpapi` is the most recent end-to-end example — read `docs/architecture/gcp-indexer-internals.md` first; it documents every piece you are about to replicate. `azureapi` is the second reference. The mandate is to build native SDK indexers so CloudQuery can eventually be removed entirely. ## Architecture (what you are building) ``` scripts// ├── _cloudquery_tables.txt # parity source (CloudQuery hub table list) ├── _resource_type_overrides.yaml # hand-curated overrides └── sync__resource_type_registry.py # registry generator src/indexers/ ├── _resource_type_registry.yaml # GENERATED catalog (never hand-edit) ├── _resource_type_registry.py # read-only loader ├── _common.py # credentials + scope resolution + tag filters ├── api_normalizers.py # raw SDK/API -> CloudQuery-shaped dict ├── api_resource_types.py # generic collector + typed collectors + specs ├── api.py # orchestration loop └── test_*.py # unit tests src/enrichers/.py # PlatformHandler.parse_resource_data src/templates/-tags.yaml # tag template src/templates/-hierarchy.yaml # hierarchy template docs/architecture/-indexer-internals.md docs/authoring/indexed-resources/.md ``` ## Core principles 1. **Parity first.** Seed the registry from the upstream CloudQuery plugin's table list (use the CloudQuery hub, not the old in-container plugin version). Every CloudQuery table must resolve via the registry by canonical name/alias. 2. **CloudQuery table name is the public contract.** Generation rules reference it as `resource_type`. The native indexer normalizes payloads into the same shape so rules don't change when the backend flips. 3. **Generic collector is the workhorse**, typed collectors are enrichment. Pick the broadest first-party API for the generic pass (Azure `resources.list()`, GCP Cloud Asset Inventory, AWS Cloud Control API). 4. **Selective, generation-rule-driven discovery.** Only collect types referenced by loaded generation rules (plus the mandatory anchor), and respect Level-of-Detail scoping. 5. **Never hand-edit the generated registry YAML.** ## Workflow ``` - [ ] 1. Obtain upstream CloudQuery table list -> scripts/

/

_cloudquery_tables.txt - [ ] 2. Author the overrides YAML (type remaps, aliases, typed flags, anchor, nulls) - [ ] 3. Write the sync script (heuristic table-name -> native-type mapping) - [ ] 4. Generate the registry YAML + write the read-only loader - [ ] 5. Add SDK deps to pyproject + regenerate lock in python:3.14-slim - [ ] 6. Write

_common.py (credentials, scope/LOD, tag filters) - [ ] 7. Write

api_normalizers.py (raw -> CloudQuery-shaped dict) - [ ] 8. Write

api_resource_types.py (generic collector + typed + specs) - [ ] 9. Write

api.py orchestrator (phases: bootstrap -> anchors -> typed -> generic) - [ ] 10. Add/confirm the enricher PlatformHandler.parse_resource_data - [ ] 11. Add src/templates/

-tags.yaml +

-hierarchy.yaml - [ ] 12. Wire pipeline: component.py, run.sh, run.py setting, cloudquery skip - [ ] 13. Tests: registry + normalizers + selective - [ ] 14. Docs: indexer-internals + indexed-resources + READMEs - [ ] 15. Run the full indexer test suite ``` ### 1–4. Registry pipeline Mirror `scripts/gcp/`: - **Table list**: scrape the CloudQuery hub plugin table reference into a `.txt` with a header noting source + version. This is the parity source. - **Overrides**: a YAML with native-type remaps, `aliases`, `typed_collectors`, the **mandatory anchor** type (e.g. GCP `gcp_projects`, Azure `azure_resources_resource_groups`), and `null` native types for tables with no API equivalent. - **Sync script**: heuristic `__` -> `/`, plus override application. Copy `sync_gcp_resource_type_registry.py` and adapt the heuristic. - **Loader** (`_resource_type_registry.py`): copy the GCP loader; provide `load_registry`, `find`, and a `find_by__type` lookup. ### 6–9. Indexer modules - **`_common.py`**: resolve credentials (K8s secret, inline key, ADC / env) and the account/project/subscription scope; provide `has_included_tags` / `has_excluded_tags` label filters. - **`api_normalizers.py`**: convert raw SDK/API objects to the CloudQuery-shaped dict the handler expects; hoist the full payload, map cloud labels/tags to `tags`, normalize names/IDs/regions. Include a `make__resource_data` helper for the synthesized anchor. - **`api_resource_types.py`**: a `*ResourceTypeSpec` dataclass, the generic collector over the broad API, typed collectors (lazy-import SDK clients), `_TYPED_COLLECTORS` keyed by canonical table name, and `find_spec` / `find_spec_by__type`. - **`api.py`** `index(ctx)` phases: 1. Bootstrap: check `IndexerBackend`, resolve creds + scope, mirror into the enricher, resolve LOD (skip scopes with LOD `none`). 2. Phase 0: emit the synthesized **anchor** resource first so children link to it at parse time. 3. Phase 1: typed collectors for accessed types that have one. 4. Phase 2: one generic pass per scope, filtered to accessed native types not already covered by a typed collector. Every resource flows: normalize -> tag filter -> `handler.parse_resource_data` -> `writer.add_resource`. ### 10–11. Handler + templates - Confirm/extend `src/enrichers/.py` `parse_resource_data` consumes the normalized dict and links children to the anchor. - Add `src/templates/-tags.yaml` and `-hierarchy.yaml` following the **`template-tags-hierarchy` rule** exactly (hierarchy ends with `resource_name`; emit `platform`, `resource_type`, `resource_name`, `child_resource`; specificity-ordered `resource_name`; dedup loop). ### 12. Pipeline wiring - `src/component.py`: add `"api"` to the `INDEXER` stage. - `src/run.sh`: include `api` in `COMPONENTS`. - `src/run.py`: coalesce `IndexerBackend` from `workspaceInfo` or `WB__INDEXER_BACKEND` into request data. - `src/indexers/cloudquery.py`: skip this platform when the native backend is selected (mutual exclusion). ### 13–14. Tests + docs (required) - Tests mirroring `test_gcp*.py`: registry loader contract, normalizer + handler round-trip, selective discovery + generic-pass filter dispatch. - `docs/architecture/-indexer-internals.md` (link it from `docs/architecture/README.md`). - `docs/authoring/indexed-resources/.md` documenting both backends, the matchable types, and the stable generation-rule contract. - Update top-level `README.md` / docs indexes if the platform is newly listed. ### 15. Verify ```bash cd src && python -m unittest discover -s indexers -p 'test_*.py' python scripts//sync__resource_type_registry.py --dry-run ``` The sync dry-run must report 0 drift (registry matches overrides + table list).