Python Recipe Runnability Test Generator

Generate Python Runnability Test

Use this skill to create the tests/test_runnability.py file that every Python recipe under core/python/, contrib/python/, or plugins/<vertical>/<solution>/ must ship (see python-validate-recipe.yml Check 4). The generated test is deliberately minimal — it just verifies the agent module imports and defines the expected globals. Business-logic testing lives elsewhere.

What This Skill Does

Runs scripts/generate_runnability_test.py against a recipe directory. Steps:

1. Locate agent.py. Walks the recipe safely (excludes .venv, venv, env, build, dist, __pycache__, node_modules, tests, *.egg-info, and dot-directories) and picks the shallowest agent.py match. If none is found, errors out and suggests --agent-file. Only one file is generated per invocation.

2. Parse agent.py AND every ancestor package __init__.py with ast to detect:

   • Top-level assignments (agent.py only, per convention) — is root_agent = ... present? Is app = ... present?
   • Import-time calls (agent.py + every ancestor __init__.py) — does vertexai.init(...) fire at module load? Does google.auth.default()? Every ancestor __init__.py is checked because Python runs each of them in order when the test does import a.b.agent (a/__init__.py, then a/b/__init__.py, then the module), so a side effect in any of them matters as much as one in agent.py. Historical bug closed by this: cross-session-memory has _, project_id = google.auth.default() in __init__.py; before, the scanner missed it and the generated test crashed in CI without ADC. Detection uses ast.walk (any depth), so it is intentionally broad — a call nested in a function body still flags the recipe; the resulting patch is a harmless no-op if it never fires at import time, whereas a missed import-time call would crash the generated test.
   • Env-var access (agent.py + every ancestor __init__.py) — does the code read GOOGLE_CLOUD_PROJECT via os.getenv / os.environ.get / os.environ["…"]? Only reads count: an os.environ["…"] = value write means the recipe sets its own value and doesn't depend on the test providing one, so it's ignored.

3. Scan all source .py files in the recipe directory tree (same safe walker) for INTEGRATION_TEST env-var reads. This is a per-package convention: agent.py often calls a helper (e.g. retrievers.create_search_tool) at module load, and THAT helper — which may live anywhere in the recipe tree, not necessarily beside agent.py — is what reads INTEGRATION_TEST. Restricting the scan to agent.py would miss it.

4. Emit the test. Two shapes:

   • Minimal (no side effects detected) — module-level import <module> + assert root_agent is not None (and app is not None if present).
   • Guarded (any side effect detected) — env-var setdefault calls at the top of the test function; a with patch(...): block around the import listing every patch needed (patch("vertexai.init") when vertexai is used, patch("google.auth.default", return_value=(MagicMock(), "test-project")) when the recipe touches google.auth); and assertions outside the with block (the patches are only needed during import; keeping them active around assertions would be misleading).

   The google.auth.default patch is what makes the test survive a recipe that calls google.auth.default() unconditionally at import time (like cross-session-memory's __init__.py). Just setting GOOGLE_CLOUD_PROJECT isn't enough for that pattern — the call still fires and still needs valid ADC — hence the patch.

   Emission is post-processed through ruff format when available, so multi-patch with (...): blocks come out already wrapped per the repo's ruff config.

5. Check that the import can actually resolve. The generated test does import <module_name>, which only works if the RECIPE ROOT is on sys.path. That is not automatic — under pytest's default prepend import mode only the test file's own directory (<recipe>/tests) is inserted. The recipe root gets there in one of these ways, reported as import_support:

   import_support	Meaning
   installable	pyproject.toml declares a [build-system], so uv sync installs the project and the package is importable.
   pythonpath-ini	[tool.pytest.ini_options].pythonpath includes ".".
   existing-conftest	A conftest.py sits at the recipe root (sufficient whatever it contains — pytest puts each conftest's own directory on sys.path, and for that one it IS the recipe root), or a tests/conftest.py that demonstrably extends sys.path.
   generated-conftest	None of the above held, so the skill wrote tests/conftest.py with a path shim.

   A tests/conftest.py is judged by AST, not text search: it counts only if it really touches sys.path, so a comment or docstring that merely mentions sys.path cannot wrongly certify it.

   Historical bug closed by this: the recipe root being importable was assumed rather than checked. A recipe with no [build-system] — common for vertical plugins under plugins/, where code lives in a plain scripts/ directory rather than an installed package — got a test that always died with ModuleNotFoundError, while prepare-python-recipe's py_compile verification still reported success.

   Adding a [build-system] is the better fix; the generated conftest says so and tells the maintainer to delete it once they do.

6. Write it to <recipe-dir>/tests/test_runnability.py (creating tests/ if needed), plus tests/conftest.py when step 5 called for it. Refuses to clobber an existing file unless --overwrite is passed.

Edit safety

• No files outside the target recipe directory are read (beyond the recipe's own .py files) or written.
• Existing tests/test_runnability.py is never silently overwritten. The user must explicitly opt in with --overwrite.
• An existing tests/conftest.py is never overwritten, not even with --overwrite — the skill reports conftest_action: skipped and warns instead. A conftest it did not write may do something clobbering would break.
• tests/ directory is created if missing (mkdir -p equivalent). No other directory or file is added.
• Ruff-clean by construction — the generated file passes ruff check and ruff format --check under the root config.