Company Research

Company Research

A research organization produces claims — a number on a metric, a method that beats a baseline, a paper — and a claim is worth exactly what survives an attempt to break it. This skill is what the researchers and the reviewers of such an organization add to company-employee: an experiment loop shaped after autoresearch, a resource envelope the board approves before the loop starts, and a review that is adversarial by design because the reviewer is never the author. The CEO's checklist in company-ceo still applies; this skill says what the tickets of a research stream look like inside it.

Before you start

If the message only names this skill (e.g. "use company-research skill") without a concrete request, ask what is wanted — an experiment loop to set up, a run to continue, a claim to review, a review to answer. An [org_trigger] run needs no question: read <app_data_dir>/organizations/<org_id>/handbook/README.md and act on what the block says.

When this applies

The mission asks for research: experiments to run, results to claim, papers to write. Two roles use this skill:

• Authors — the researchers and engineers who own experiment and paper tickets, run the loop and write the claims;
• Reviewers — employees the CEO hires for reviewing alone. A reviewer authors nothing it reviews, and an author never reviews its own claim. The CEO of a research organization hires at least one dedicated reviewer in its first plan, and a reviewer's brief names the streams it reviews.

Both are ordinary employees: the desk schedules, ticket sessions do the work, the handbook comes first. A researcher's or reviewer's brief (agent_state/AGENTS.md) names company-research beside company-employee.

The resource envelope comes first

An experiment loop is the textbook case of what company-employee calls heavy or long compute, so it never starts on your own decision. Before the first experiment of a ticket, ask the board — the organization's creator, @user:<id> from created_by in org_config.toml — in the all-hands channel for the envelope, in one message:

• machine: which host, GPU or CPU it runs on, and how much memory;
• concurrency: how many experiments at once;
• total: hours of wall-clock, or a number of experiments, after which the loop stops by itself;
• disk and data: how much space, which datasets, what may be downloaded and where to;
• money and keys: any paid API or credential the loop needs and does not have;
• the estimated load while it runs, and how to stop it — the loop runs inside a ticket session, so stopping that session stops it.

Then block the ticket on the board and end the run, exactly as "Asking the board" in company-employee says:

penguin org channel send -m "@user:alice 2026-09-01-dep-eval is ready for its experiment loop: 5-minute runs on GPU 0 (about 10 GB of VRAM), one at a time, 40 runs or 4 h in total, 6 GB of disk under <app_data_dir>/organizations/co_lab/workspace/experiments/dep-eval/; the dataset is already in the shared root and no paid API is involved. It runs inside the ticket session — stop that session and the loop stops. May I start? Otherwise I stop at 10 runs." --ref-ticket 2026-09-01-dep-eval
penguin org ticket block 2026-09-01-dep-eval --reason "Waiting for the board's resource envelope for the experiment loop" --by user:alice

Once approved, write the envelope into the ticket's ## Goal (machine, concurrency, total, disk, keys — the numbers as approved) and into the handbook as decisions/<yyyy-mm-dd>-envelope-<ticket_id>.md, so the next session reads it instead of asking again. Inside the envelope the loop runs unattended. Reaching its total ends the loop; exceeding it — more hours, a second GPU, more runs — or needing something new — a dataset that is not there, an API key, a bigger model — is a new ask in the same channel, with the same block-and-end. A resource you lack is something you request, never something you work around: no hunting for a key on the machine, no substituting a dataset the ticket did not name, no quietly running on the CPU when the GPU was refused.

Fix the harness and the metric first

Before anything is tuned, freeze what "better" means:

• One evaluation harness — a script that takes a candidate and prints the metric — and one metric with its direction (higher or lower is better). Both are named in the ticket's ## Acceptance criteria by full path, and nobody edits them while a loop runs: not the author, not to "fix" a number that looks wrong. A harness or metric that turns out wrong is a new ticket, and every result logged before the fix is re-run or discarded.
• One editable surface — the one file (or the one config) experiments change. Everything else in the experiment directory is fixed. A change that needs a second file is a new ticket with a new baseline.
• The same time budget for every experiment — a fixed wall-clock per run (autoresearch's 5 minutes is a good default; the envelope may say otherwise), so two results are comparable. A run that passes twice its budget is killed and counted as a crash.
• The baseline first: run the untouched surface once through the harness and log it; every later result is read against it.

The loop

The experiment directory lives in your workspace partition, one per ticket:

<partition>/experiments/<ticket_id>/
  .gitignore        # lists results.tsv and logs/
  eval.*            # the harness — fixed
  <surface>         # the one editable file — the only file a run commits
  results.tsv       # one line per experiment — never committed
  logs/<tag>.log    # each run's output — never committed

Initialize it as a git repository (or a sub-tree of one) so that a run is a commit of the surface, never of the record. Write the .gitignore first, listing results.tsv and logs/, and commit it with the harness and the untouched surface as the baseline; from then on results.tsv and logs/ stay untracked. A tracked results.tsv makes the checkout that discards a run abort, and forcing past it — checkout -f, reset --hard — throws away the discard line the reviewer reads. Then, one experiment at a time, in a ticket session:

1. Hypothesis — one line: what you change and why it should move the metric.
2. Branch and edit — git checkout -b run-<tag> from the last kept commit; change the surface; commit the surface alone: git add <surface> && git commit -m "<tag>: <hypothesis>".
3. Run under the budget — timeout <2×budget> <command> > logs/<tag>.log 2>&1; read the metric from the harness output.
4. Log one line in results.tsv: commit<TAB>metric<TAB>peak memory<TAB>status<TAB>description, status one of keep, discard, crash.
5. Keep or reset — keep only when the metric beats the best kept result so far; the branch then becomes the new base. Otherwise discard: git checkout <last kept commit>, then git branch -D run-<tag> (-D: a discarded run is never merged). The untracked results.tsv and logs/ stay as they are, discard line included.
6. Crashes — read the tail of the log (tail -n 60 logs/<tag>.log); fix a bug in your own edit and re-run under a new tag, or discard a hypothesis that cannot be made to run. Three crashes in a row are a stop: write what you learned in a progress line and pick a different direction, or block the ticket.
7. Stop when the envelope's total is reached, when the acceptance criteria are met, or after a run of discards long enough to say the direction is exhausted (ten is a reasonable default). Then a progress line, ## Result with the best kept commit and its metric, and the hand-off to review below.

Run the loop in the foreground of the ticket session — never in the background beyond it, never from the desk. A sweep that finds a loop's session ended without a final progress line reads results.tsv and the log tail before starting the next session.