Rocky Test Fixtures

Dagster test fixture regeneration

The dagster integration has two parallel sources of test data with different roles. Conflating them causes confusion, so this skill is worth reading before touching either.

The two test-data surfaces

Tests in test_types.py, test_translator.py, test_component.py, etc. consume the *_json fixtures backed by scenarios.py. The generated corpus is exercised solely by test_generated_fixtures.py to catch shape drift between the Rust binary and the Pydantic models.

A legacy tests/fixtures/*.json directory previously held the parsing-test corpus; it was removed once scenarios.py became the source of truth. See conftest.py for the migration note. The --in-place flag in scripts/regen_fixtures.sh still targets that directory and is effectively a no-op today.

When to use this skill

• Changing any *Output struct that affects a fixture's shape (pairs with the rocky-codegen skill — codegen updates the Pydantic models; the captured fixtures should still parse against them)
• Adding a brand new CLI command that needs a parsing test in test_types.py (you'll add a scenarios.py entry, not a JSON file)
• Investigating why test_generated_fixtures.py fails after a binary or schema change
• Debugging non-determinism in a captured fixture (timestamps, durations, row counts)

Regeneration workflow

just regen-fixtures
# or
./scripts/regen_fixtures.sh

Writes to integrations/dagster/tests/fixtures_generated/. The captured JSON should git diff cleanly except for fields that legitimately changed; non-trivial diffs warrant inspection.

After regenerating, always run from integrations/dagster/:

uv run pytest tests/test_generated_fixtures.py -v

This catches any captured-output shape that no longer parses against the current Pydantic models.

Prerequisites

The script hard-fails if either is missing:

1. Release binary at engine/target/release/rocky — build with:

   cd engine && cargo build --release --bin rocky

   or run just codegen (which also builds in release mode and shares the artifact).

2. duckdb CLI on $PATH — used to seed the playground POC's DuckDB file:

   brew install duckdb        # macOS

Two POCs back the corpus

regen_fixtures.sh runs the binary against two playground POCs, captured into different subdirectories of fixtures_generated/:

If you need a fixture shape that neither POC produces naturally (e.g. drift.json from a non-zero schema diff), add it as a scenarios.py entry rather than coercing the POC into producing it.

Determinism: why captured JSON is stable across runs

Without post-processing, every regen would produce a slightly different diff because Rocky stamps:

• Duration fields with real wall-clock elapsed time (_ms, _seconds, _secs suffixes)
• Timestamp fields with now() (updated_at, started_at, finished_at, timestamp, captured_at)

The capture helper inside regen_fixtures.sh runs a Python normalizer (scripts/_normalize_fixture.py) over each captured JSON file that:

• Zeroes any numeric field whose key ends in _ms / _seconds / _secs
• Replaces any string field whose key is in the wall-clock set with "2000-01-01T00:00:00Z"

Important invariants the normalizer preserves:

• last_value (watermarks) — NOT touched. It's a logical value derived from seeded data and is genuinely deterministic across runs.
• Anything inside a data payload — NOT touched. Only top-level wall-clock fields get the sentinel.

If you add a new timing or timestamp field to an *Output struct, you may need to extend TIMING_SUFFIXES or WALL_CLOCK_FIELDS in the normalizer. Otherwise your new fixture will drift on every regen.