Optimize Nethack Policy

Optimize an NLE NetHack Policy

Maximize mean shaped return in deterministic but hidden NetHackScore Episodes. The task uses NLE 1.3.0, NetHack 3.6.7, a fixed neutral human male Monk, the standard 23-action task set, and at most 5,000 Policy steps. Each Policy failure receives a large negative return, so validity and termination come first.

Respect the ABI

• Export make_policy(context) from policy.py and return an object with act(observation).
• Return an exact integer from 0 through 22. Booleans, floats, containers, and out-of-range integers are invalid; the Benchmark never repairs an Action.
• A fresh Policy process and instance are created for every Episode. Keep map, inventory, prompt, and plan state only between act() calls in that Episode.
• Read public static configuration from context.environment_parameters. Never infer, request, or search for Environment seeds or Host paths.

Actions

 0 more               8 northwest         16 run_northwest
 1 north              9 run_north         17 up
 2 east              10 run_east          18 down
 3 south             11 run_south         19 wait
 4 west              12 run_west          20 kick
 5 northeast         13 run_northeast     21 eat
 6 southeast         14 run_southeast     22 search
 7 southwest         15 run_southwest

Actions are NLE keyboard inputs. In a prompt, the same raw key can mean an answer rather than movement. For example Actions 1 through 8 emit k l j h u n b y; Action 21 emits e, Action 22 emits s, and Action 0 emits Enter. Inspect message and input_mode before interpreting an Action as an ordinary command.

Observation

The observation is a dictionary:

• screen.glyphs: int16 TensorValue, shape (21, 79);
• screen.chars: uint8 TensorValue, shape (21, 79), row-major visible character codes;
• screen.colors: uint8 TensorValue, shape (21, 79);
• stats: named public NLE blstats, including x, y, score, HP, depth, gold, experience, turn, hunger, encumbrance, dungeon level, and conditions;
• message: the current public NetHack message;
• inventory: only populated entries with letter, description, glyph, and object_class;
• input_mode: normal, yes_no, get_line, or more.

The Policy does not receive NLE's privileged internal array, raw seeds, ttyrec paths, rewards, or evaluator state. Decode tensors from TensorValue.data; do not import or reach into the trusted NLE Environment.

Build an Episode-local controller

Use explicit layers that can be tested separately:

1. Prompt handler: respond to <More>, direction, eating, and yes/no questions before normal navigation. A prompt that is mistaken for movement commonly creates frozen-step penalties or loops.
2. Map memory: combine visible characters, colors, glyphs, current position, and dungeon level. Track visited, blocked, dangerous, door, corridor, staircase, item, and monster cells without assuming a hidden seed route.
3. Immediate safety: monitor HP, hunger, conditions, nearby monsters, and escape space. Do not start a long run or repeated search while threatened.
4. Exploration: prefer reachable low-visit frontiers, search near plausible dead ends, handle doors deliberately, and avoid immediate reversal loops.
5. Resource state: track public inventory descriptions and observed outcome messages. Initiating eat and selecting an inventory letter are separate decisions.
6. Progress plan: descend when prepared, collect useful public items, and balance score gain against survival and frozen-step penalties.

The standard 23-action profile is deliberately narrower than a full NetHack keyboard. Do not write plans that require unavailable commands such as wield, wear, quaff, read, open, or arbitrary inventory letters.

Optimize the actual metric

The scalar score is mean Episode return from NetHackScore-v0: NetHack score deltas plus a -0.01 penalty when repeated Environment steps do not advance game time. mean_game_score, depth, deaths, ascensions, truncations, and Policy failures are secondary diagnostics. A Policy failure receives -max_episode_steps, not the partial return accumulated before failure. frozen_steps, mean_frozen_steps, frozen_step_fraction, and mean_frozen_penalty expose the exact aggregate contribution of unchanged-turn Actions to this score; they do not select or summarize trajectory content.

Small batches have high procedural variance. First eliminate failures and frozen prompt loops, then compare unchanged candidates on equal larger batches. Use 1--4 Episodes per search submission until memory and runtime are measured.