Refresh Source

Refresh a verified source

Arguments: $ARGUMENTS — first token is the source folder name under sources/, anything after -- is user hints (specific user reports, suspected areas).

Work through the phases in order. Do not skip phase 0 — wrong conclusions are usually formed by analyzing code that is no longer alive or pins that no longer reflect reality.

Phase 0 — Ground truth

1. Read everything in sources/<name>/, sources/<name>_pipeline.py, tests/<name>/.
2. git log --oneline -- sources/<name> — look for past removals/refactors. Hunt dead code: files, fixtures, conftest helpers, docstrings and README claims orphaned by earlier PRs. Dead code should be deleted, never "fixed". Dead code may also be the only reason for a heavy dependency — deleting it can drop whole packages from requirements.txt.
3. Compare the THREE dependency declarations and flag mismatches:
   • sources/<name>/requirements.txt — what dlt init users actually install (often unbounded!)
   • pyproject.toml dependency group <name> — what CI tests
   • any version pinned in code (e.g. api_version = "...") A CI group pinned years behind an unbounded requirements.txt means CI is testing a different client than users get — that is how breakage ships silently.
4. Check open/closed GitHub issues for the source (gh search issues) and any user hints from the arguments.

Phase 1 — Research the upstream (web search, authoritative sources only)

Use WebSearch/WebFetch against official docs, SDK changelogs, and SDK migration guides (vendor docs site, vendor GitHub wiki/CHANGELOG). For each finding record the source URL.

• Current SDK major version and every breaking change between the version CI pins and today (migration guides are gold: object model changes, removed dict-inheritance, auth patterns, deprecated module-level clients).
• Current API version / versioning model; what the code pins vs. what is current; breaking changes in between (renamed/removed/restructured fields the source or its helpers depend on).
• Official request/response shapes the code touches: list envelopes, pagination cursors, filter parameter formats, retention limits (e.g. event retention windows). Capture documented example payloads — they become test fixtures in phase 4.

Phase 2 — Reproduce empirically. Never claim a breakage you haven't run.

Use ephemeral environments to prove each suspected breakage and each proposed fix, at both ends of the allowed dependency range:

cd /tmp && uv run --with "<sdk>==<oldest-allowed>" python - <<'EOF' ... EOF
cd /tmp && uv run --with "<sdk>==<newest>" python - <<'EOF' ... EOF

Construct real SDK objects (e.g. construct_from-style factories) and exercise the exact code path that is suspected broken. Subtle version differences matter (e.g. a conversion helper being shallow in one major and recursive in another) — only execution reveals them.

Phase 3 — Plan (present before implementing)

Tier the findings; keep tiers separable:

• P0 — unbreak: minimal fixes that work across the whole supported dependency range; align CI group with what fresh users install; fix requirements bounds (add upper bound to majors not yet verified). Must NOT change loaded table schemas.
• P1 — modernize client: new SDK client patterns, drop global state, configurable API version. Anything that changes response shapes changes loaded schemas for existing users — call this out explicitly as breaking and keep it out of P0.
• P2 — features: new endpoints, incremental improvements.

Prefer dlt built-ins over hand-rolled code (dlt.common.time.ensure_pendulum_datetime, dlt.common logger, rest_api helpers) — but verify the utility exists in the oldest dlt allowed by the source's requirements.txt before using it.