Rl Env From Description

RL Env From Description

Convert a plain-English description of an RL training environment into runnable code across OpenEnv, OpenReward (ORS), Verifiers, and NeMo Gym. Two other framework variants (SkyRL Gym, GEM) are secondary and only relevant for text-action-with-tag-parsing envs — produce them only if the user asks.

When to use

• A user describes an env in plain English (a goal, an action surface, or a reward shape) and wants code.
• A user asks to "build an env for X", "scaffold an RL env", "port this env to OpenEnv/ORS/Verifiers/NeMo Gym".
• A user already has a runnable env in one framework and wants the same env in others.
• A user asks "what's the right way to design my reward / state / tool surface for this task" — start with the interview below, then implement.

Do not use for: training runs (TRL/GRPO config), evaluation harness work, or general agent-design questions that don't end with new env code.

Recommended layout (suggest, don't impose)

A clean shape that scales well — but the user gets to pick the actual paths:

<env_dir>/                          # whatever the user names it
├── <domain>.py                     # SHARED pure logic (e.g. game.py)
├── tasks.py                        # SHARED list of task dicts (optional)
├── openenv/                        # OpenEnv variant (HTTP, MCP)
├── ors/                            # ORS variant (HTTP, REST + SSE)
├── verifiers/                      # Verifiers variant (in-process)
└── nemo_gym/                       # NeMo Gym variant (HTTP, REST + cookies)

Inside each framework folder, the public contract is:

• pyproject.toml — framework-specific deps
• __init__.py
• One implementation file (server.py for ORS, server/<env>_environment.py for OpenEnv, env.py for Verifiers, server.py for NeMo Gym)
• rollout.py — runs an LLM against the env end-to-end
• README.md — one-page consumption guide

Always ask the user where they want files written. If they don't have a preference, propose the layout above. Don't force it.

The four-step flow

Step 1 — Interview the user (focused, not exhaustive)

Ask only the questions that determine architecture. The full bank lives in references/interview.md; the must-cover set is:

1. What does the agent DO? One sentence describing the goal and the loop.
2. Action surface — structured tool calls (most cases) or free text with tag parsing (rare; only when the model has no tool-calling support).
3. State — does anything persist across turns? Per-session sandbox? In-memory dict? Nothing?
4. Reward — when does it fire (per-step, on terminate, post-episode)? What's the success criterion?
5. External backends — sandbox (E2B), web service, none?
6. Termination — fixed turn cap, model-emits-terminate, or a derived condition.
7. Where should the files live? Project-relative path; never assume.

If the user has already given enough signal in their description (e.g. they cited an existing env they want to mirror), skip questions whose answers are obvious. Don't make people repeat themselves.

When in doubt about an architectural choice, propose a default with a one-line rationale and let the user veto.

Step 2 — Pick the closest archetype

Match the user's description to one of these archetypes; tell them which archetype you're using and why:

Step 3 — Implement in dependency order

The shared module first, then per-framework variants. Order doesn't matter between frameworks.

1. <env_dir>/<domain>.py + <env_dir>/tasks.py — the only file that contains domain logic. Frameworks just wrap it. Keep it pure-Python; no framework imports.
2. OpenEnv — read references/openenv.md (planner-level) or trigger generate-openenv-env skill (full workflow). Use MCPEnvironment + @mcp.tool + create_app(...) in server/app.py.
3. ORS — read references/ors.md or trigger generate-ors-env. Use Environment + @tool methods + ToolOutput(blocks=[...], reward=..., finished=...). Per-tool-call reward is the framework's defining feature.
4. Verifiers — read references/verifiers.md or trigger generate-verifiers-env. Plain Python tool functions on a toolkit class; vf.ToolEnv + vf.Rubric for native consumption.
5. NeMo Gym — read references/nemo_gym.md or trigger generate-nemo-gym-env. SimpleResourcesServer with one app.post("/<tool>") per tool; cookie sessions; post-episode /verify reward.