Agent skill

torchforge RL Training

by Orchestra-Research in Orchestra-Research/AI-Research-SKILLs

Guides reinforcement-learning research with torchforge, Meta's PyTorch-native library that keeps RL algorithms apart from infrastructure, including GRPO math-reasoning runs.

MITAuto-check passedAI & LLM Engineering

Install torchforge RL Training

skills CLI
$ npx skills add Orchestra-Research/AI-Research-SKILLs --skill torchforge-rl-training -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install Orchestra-Research/AI-Research-SKILLs torchforge-rl-training --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/Orchestra-Research/AI-Research-SKILLs.git skills-src && mkdir -p .claude/skills && cp -r skills-src/06-post-training/torchforge .claude/skills/torchforge-rl-training && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
torchforge-rl-training
GitHub stars
13k
Used in
2 other repos
Token cost
~2.5k tokens
SKILL.md length
368 words
Files
3 (incl. references)
Skills in repo
96
Repo updated
First seen
Licence
MIT

At a glance

Guides reinforcement-learning research with torchforge, Meta's PyTorch-native library that keeps RL algorithms apart from infrastructure, including GRPO math-reasoning runs.

  • Works in 6 steps: Create Configuration → Define Reward Function → Launch Training → …
  • Experimenting with new RL algorithms without rewriting infrastructure
  • SKILL.md covers When to Use torchforge, Key Features, Architecture Overview and Installation, plus 7 more sections
  • Calls python and conda

What it does

torchforge splits reinforcement learning into algorithm code that you write and infrastructure it handles for you: distributed training, inference and weight sync. It has no Ray dependency, scales through the Monarch actor system, uses TorchTitan for model parallelism, vLLM for inference and TorchStore for syncing, and ships GRPO, DAPO, CISPO, GSPO and SAPO loss functions. The skill covers conda and ROCm installation plus quick-start commands for supervised fine-tuning and GRPO using YAML configs.

Its main workflow trains a math reasoning model with GRPO. The checklist asks for three or more GPUs, split between the trainer, a reference model and the generator, a model from the Hugging Face Hub and a dataset such as GSM8K or MATH. Steps then cover writing the config and defining rewards with the MathReward and ThinkingReward helpers. The skill warns that torchforge is experimental and directs you to miles or verl for stability. The excerpt is cut off at the launch step.

When your agent uses it

  • Experimenting with new RL algorithms without rewriting infrastructure
  • Running GRPO training on a math dataset with a custom reward
  • Wanting PyTorch-native RL tooling instead of a Ray-based stack
  • Scaling an RL experiment with Monarch and TorchTitan

Example prompts

  • “Set up torchforge in a new conda environment and run the GRPO quick start config.”
  • “Write a GRPO config for Qwen2.5 7B Instruct trained on GSM8K with a math reward.”
  • “Explain how torchforge divides the trainer, reference model and generator across three GPUs.”
  • “Is torchforge stable enough for production, or should we use verl instead?”

Requirements

  • A conda environment with torchforge installed
  • Three or more GPUs for GRPO training
  • A model from the Hugging Face Hub and a training dataset

Workflow steps

6 steps, taken from the step headings in SKILL.md.

  1. Create Configuration
  2. Define Reward Function
  3. Launch Training
  4. Monitor Progress
  5. Create Loss Class
  6. Integrate into Application

What it can do on your machine

Read from SKILL.md and the folder at commit 773a529. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Shell commands in SKILL.md call:

    • python
    • conda

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • github.com
    • meta-pytorch.org
    • discord.gg

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

torchforge RL Training loads about 2.5k tokens when it runs, and up to ~6.3k if it reads all its reference files. Until then it costs about 65 tokens; SKILL.md has 368 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~65
When it runs · the whole SKILL.md, loaded when a task matches
~2.5k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~6.3k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Orchestra-Research/AI-Research-SKILLs at commit 773a529, republished under its MIT licence (© Orchestra-Research). 368 words, ~2,489 tokens.

Download SKILL.mdSave it as .claude/skills/torchforge-rl-training/SKILL.md (or your agent's skills folder). This skill also uses 2 other files; get the full folder from GitHub.
name
torchforge-rl-training
description
Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.
version
1.0.0
author
Orchestra Research
license
MIT
tags
Reinforcement Learning, PyTorch, GRPO, SFT, Monarch, TorchTitan, Meta
dependencies
torch>=2.9.0, torchtitan>=0.2.0, vllm, monarch

torchforge: PyTorch-Native Agentic RL Library

torchforge is Meta's PyTorch-native RL library that separates infrastructure concerns from algorithm concerns. It enables rapid RL research by letting you focus on algorithms while handling distributed training, inference, and weight sync automatically.

When to Use torchforge

Choose torchforge when you need:

  • Clean separation between RL algorithms and infrastructure
  • PyTorch-native abstractions (no Ray dependency)
  • Easy algorithm experimentation (GRPO, DAPO, SAPO in ~100 lines)
  • Scalable training with Monarch actor system
  • Integration with TorchTitan for model parallelism

Consider alternatives when:

  • You need production-ready stability → use miles or verl
  • You want Megatron-native training → use slime
  • torchforge is experimental and APIs may change

Key Features

  • Algorithm isolation: Implement RL algorithms without touching infrastructure
  • Scalability: From single GPU to thousands via Monarch
  • Modern stack: TorchTitan (training), vLLM (inference), TorchStore (sync)
  • Loss functions: GRPO, DAPO, CISPO, GSPO, SAPO built-in

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│ Application Layer (Your Code)                           │
│ - Define reward models, loss functions, sampling        │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│ Forge API Layer                                         │
│ - Episode, Group dataclasses                           │
│ - Service interfaces (async/await)                      │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│ Distributed Services (Monarch)                          │
│ ├── Trainer (TorchTitan FSDP)                          │
│ ├── Generator (vLLM inference)                          │
│ ├── Reference Model (frozen KL baseline)               │
│ └── Reward Actors (compute rewards)                    │
└─────────────────────────────────────────────────────────┘

Installation

bash
# Create environment
conda create -n forge python=3.12
conda activate forge

# Install (handles PyTorch nightly + dependencies)
./scripts/install.sh

# Verify
python -c "import torch, forge, vllm; print('OK')"
ROCm Installation
bash
./scripts/install_rocm.sh

Quick Start

SFT Training (2+ GPUs)
bash
python -m apps.sft.main --config apps/sft/llama3_8b.yaml
GRPO Training (3+ GPUs)
bash
python -m apps.grpo.main --config apps/grpo/qwen3_1_7b.yaml

Workflow 1: GRPO Training for Math Reasoning

Use this workflow for training reasoning models with group-relative advantages.

Prerequisites Checklist
  • 3+ GPUs (GPU0: trainer, GPU1: ref_model, GPU2: generator)
  • Model from HuggingFace Hub
  • Training dataset (GSM8K, MATH, etc.)
Step 1: Create Configuration
yaml
# config/grpo_math.yaml
model: "Qwen/Qwen2.5-7B-Instruct"

dataset:
  path: "openai/gsm8k"
  split: "train"
  streaming: true

training:
  batch_size: 4
  learning_rate: 1e-6
  seq_len: 4096
  dtype: bfloat16
  gradient_accumulation_steps: 4

grpo:
  n_samples: 8           # Responses per prompt
  clip_low: 0.2
  clip_high: 0.28
  beta: 0.1              # KL penalty coefficient
  temperature: 0.7

services:
  generator:
    procs: 1
    num_replicas: 1
    with_gpus: true
  trainer:
    procs: 1
    num_replicas: 1
    with_gpus: true
  ref_model:
    procs: 1
    num_replicas: 1
    with_gpus: true
Step 2: Define Reward Function
python
# rewards.py
# Reward functions are in forge.data.rewards
from forge.data.rewards import MathReward, ThinkingReward
import re

# Or define your own reward function
class CustomMathReward:
    def __call__(self, prompt: str, response: str, target: str) -> float:
        # Extract answer from response
        match = re.search(r'\\boxed{([^}]+)}', response)
        if not match:
            return 0.0

        answer = match.group(1).strip()
        return 1.0 if answer == target else 0.0
Step 3: Launch Training
bash
python -m apps.grpo.main --config config/grpo_math.yaml
Step 4: Monitor Progress
  • Check W&B dashboard for loss curves
  • Verify entropy is decreasing (policy becoming more deterministic)
  • Monitor KL divergence (should stay bounded)

Show full SKILL.md (142 more words)Show less

Workflow 2: Custom Loss Function

Use this workflow to implement new RL algorithms.

Step 1: Create Loss Class
python
# src/forge/losses/custom_loss.py
import torch
import torch.nn as nn

class CustomLoss(nn.Module):
    def __init__(self, clip_range: float = 0.2, beta: float = 0.1):
        super().__init__()
        self.clip_range = clip_range
        self.beta = beta

    def forward(
        self,
        logprobs: torch.Tensor,
        ref_logprobs: torch.Tensor,
        advantages: torch.Tensor,
        padding_mask: torch.Tensor,
    ) -> torch.Tensor:
        # Compute importance ratio
        ratio = torch.exp(logprobs - ref_logprobs)

        # Clipped policy gradient
        clipped_ratio = torch.clamp(
            ratio,
            1 - self.clip_range,
            1 + self.clip_range
        )
        pg_loss = -torch.min(ratio * advantages, clipped_ratio * advantages)

        # KL penalty
        kl = ref_logprobs - logprobs

        # Apply mask and aggregate
        masked_loss = (pg_loss + self.beta * kl) * padding_mask
        loss = masked_loss.sum() / padding_mask.sum()

        return loss
Step 2: Integrate into Application
python
# apps/custom/main.py
from forge.losses.custom_loss import CustomLoss

loss_fn = CustomLoss(clip_range=0.2, beta=0.1)

# In training loop
loss = loss_fn(
    logprobs=logprobs,
    ref_logprobs=ref_logprobs,
    advantages=advantages,
    padding_mask=padding_mask,
)

Workflow 3: Multi-GPU Distributed Training

Use this workflow for scaling to multiple GPUs or nodes.

Configuration for Distributed
yaml
# config/distributed.yaml
model: "meta-llama/Meta-Llama-3.1-8B-Instruct"

parallelism:
  tensor_parallel_degree: 2    # Split model across GPUs
  pipeline_parallel_degree: 1
  data_parallel_shard_degree: 2

services:
  generator:
    procs: 2                   # 2 processes for TP=2
    num_replicas: 1
    with_gpus: true
  trainer:
    procs: 2
    num_replicas: 1
    with_gpus: true
Launch with SLURM
bash
# Submit job
sbatch --nodes=2 --gpus-per-node=8 run_grpo.sh
Launch Locally (Multi-GPU)
bash
# 8 GPU setup
python -m apps.grpo.main \
    --config config/distributed.yaml \
    --trainer.procs 4 \
    --generator.procs 4

Core API Reference

Training Batch Format

torchforge uses dictionary-based batches for training:

python
# inputs: list of dicts with torch.Tensor values
inputs = [{"tokens": torch.Tensor}]

# targets: list of dicts with training signals
targets = [{
    "response": torch.Tensor,
    "ref_logprobs": torch.Tensor,
    "advantages": torch.Tensor,
    "padding_mask": torch.Tensor
}]

# train_step returns loss as float
loss = trainer.train_step(inputs, targets)
Completion

Generated output from vLLM:

python
@dataclass
class Completion:
    text: str              # Generated text
    token_ids: list[int]   # Token IDs
    logprobs: list[float]  # Log probabilities
    metadata: dict         # Custom metadata

Built-in Loss Functions

Loss Functions

Loss functions are in the forge.losses module:

python
from forge.losses import SimpleGRPOLoss, ReinforceLoss

# SimpleGRPOLoss for GRPO training
loss_fn = SimpleGRPOLoss(beta=0.1)

# Forward pass
loss = loss_fn(
    logprobs=logprobs,
    ref_logprobs=ref_logprobs,
    advantages=advantages,
    padding_mask=padding_mask
)
ReinforceLoss
python
from forge.losses.reinforce_loss import ReinforceLoss

# With optional importance ratio clipping
loss_fn = ReinforceLoss(clip_ratio=0.2)

Common Issues and Solutions

Issue: Not Enough GPUs

Symptoms: "Insufficient GPU resources" error

Solutions:

yaml
# Reduce service requirements
services:
  generator:
    procs: 1
    with_gpus: true
  trainer:
    procs: 1
    with_gpus: true
  # Remove ref_model (uses generator weights)

Or use CPU for reference model:

yaml
ref_model:
  with_gpus: false
Issue: OOM During Generation

Symptoms: CUDA OOM in vLLM

Solutions:

yaml
# Reduce batch size
grpo:
  n_samples: 4  # Reduce from 8

# Or reduce sequence length
training:
  seq_len: 2048
Issue: Slow Weight Sync

Symptoms: Long pauses between training and generation

Solutions:

bash
# Enable RDMA (if available)
export TORCHSTORE_USE_RDMA=1

# Or reduce sync frequency
training:
  sync_interval: 10  # Sync every 10 steps
Issue: Policy Collapse

Symptoms: Entropy drops to zero, reward stops improving

Solutions:

yaml
# Increase KL penalty
grpo:
  beta: 0.2  # Increase from 0.1

# Or add entropy bonus
training:
  entropy_coef: 0.01

Resources

© Orchestra-Research, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 2 other files (references) in 06-post-training/torchforge of Orchestra-Research/AI-Research-SKILLs.

  • SKILL.md
  • references/api-reference.md
  • references/troubleshooting.md

Open the folder on GitHubat commit 773a529

Used in 2 other repositories

We found 2 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 2 other GitHub owners. This page covers the copy in Orchestra-Research/AI-Research-SKILLs, which our catalogue first saw on October 7, 2026.

Compare with similar skills

torchforge RL Training next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

torchforge RL Training compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
torchforge RL Training this skillOrchestra-Research/AI-Research-SKILLs13k2 repos~2.5kAutomated safety check: PassMIT
Magpie Kernel Evaluatoramd/skills406—~2.3kAutomated safety check: PassMIT
ML Experiment IterationLeeroo-AI/superml195—~4.8kAutomated safety check: PassApache-2.0
Alphagenome Predictionsgenomicsxai/alphagenome-pytorch162—~868Automated safety check: PassApache-2.0
ML Training Run VerifierLeeroo-AI/superml195—~3.8kAutomated safety check: PassApache-2.0
Quark Torch Quant Perfamd/Quark181—~3kAutomated safety check: PassMIT

Similar skills

  • Benchmarks LLM inference and drives GPU kernel optimization with Magpie.

    406 GitHub stars~2.3k tokensUpdated today
    AI & LLM EngineeringAuto-check passed
  • ML Experiment Iteration

    Leeroo-AI/superml

    Produces ranked, evidence-grounded next steps when an ML experiment has stalled, drawing on a Leeroopedia knowledge base or on fetched docs and issues.

    195 GitHub stars~4.8k tokensUpdated 6 mo ago
    AI & LLM EngineeringAuto-check passed
  • Alphagenome Predictions

    genomicsxai/alphagenome-pytorch

    Run AlphaGenome-PyTorch to get genomic track predictions — via the agt predict CLI (single locus, BED regions, whole chromosomes, raw FASTA sequences, or per-gene count tables/AnnData), variant…

    162 GitHub stars~868 tokensUpdated 24 days ago
    AI & LLM EngineeringAuto-check passed
  • ML Training Run Verifier

    Leeroo-AI/superml

    Checks training code, configs and math against documented framework behavior before an expensive run, citing a knowledge base or official docs for every claim.

    195 GitHub stars~3.8k tokensUpdated 6 mo ago
    AI & LLM EngineeringAuto-check passed
  • Run, resume, monitor, diagnose, and report Quark Quant-Perf workflows for PyTorch and HuggingFace transformers models.

    181 GitHub stars~3k tokensUpdated 12 days ago
    AI & LLM EngineeringAuto-check passed
  • ExecuTorch Model Export

    pytorch/executorch

    Export a PyTorch model to .pte format for ExecuTorch. Use when converting models, lowering to edge, or generating .pte files.

    5.1k GitHub stars~238 tokensUpdated today
    AI & LLM EngineeringAuto-check passed

More from Orchestra-Research/AI-Research-SKILLs

All 96 skills in this repo
  • AudioCraft Audio Generation

    Orchestra-Research/AI-Research-SKILLs

    Generates music from text descriptions with MusicGen and sound effects with AudioGen, using Meta's AudioCraft PyTorch library with melody and style conditioning.

    13k GitHub starsUsed in 8 repos~3.9k tokens
    Auto-check passed
  • LLM Benchmarking with lm-evaluation-harness

    Orchestra-Research/AI-Research-SKILLs

    Runs lm-evaluation-harness to benchmark language models on academic suites such as MMLU, GSM8K and HumanEval, compare models and track training checkpoints.

    13k GitHub starsUsed in 8 repos~3k tokens
    Auto-check passed
  • Segment Anything Model Guide

    Orchestra-Research/AI-Research-SKILLs

    Guide to using Meta's Segment Anything Model for zero-shot image segmentation with point, box or mask prompts, or automatic mask generation.

    13k GitHub starsUsed in 8 repos~3.3k tokens
    Auto-check passed
  • Chroma Vector Database

    Orchestra-Research/AI-Research-SKILLs

    Shows how to store documents and embeddings in Chroma, query them by similarity with metadata filters, and persist them to disk for RAG and semantic search projects.

    13k GitHub starsUsed in 7 repos~2.3k tokens
    Auto-check passed
  • CLIP Image-Text Matching

    Orchestra-Research/AI-Research-SKILLs

    Explains OpenAI's CLIP model for zero-shot image classification, image-text similarity, semantic image search and content moderation, with install steps and code patterns.

    13k GitHub starsUsed in 7 repos~1.7k tokens
    Auto-check passed
  • Whisper Speech Recognition

    Orchestra-Research/AI-Research-SKILLs

    Transcribes audio with OpenAI's Whisper: 99 languages, translation to English, language detection, six model sizes and word-level timestamps, from Python or the CLI.

    13k GitHub starsUsed in 7 repos~1.9k tokens
    Auto-check: notes

Questions about torchforge RL Training

What does torchforge RL Training do?

Guides reinforcement-learning research with torchforge, Meta's PyTorch-native library that keeps RL algorithms apart from infrastructure, including GRPO math-reasoning runs. torchforge splits reinforcement learning into algorithm code that you write and infrastructure it handles for you: distributed training, inference and weight sync. It has no Ray dependency, scales through the Monarch actor system, uses TorchTitan for model parallelism, vLLM for inference and TorchStore for syncing, and ships GRPO, DAPO, CISPO, GSPO and SAPO loss functions.

When should I use torchforge RL Training?

torchforge RL Training fits situations like: experimenting with new RL algorithms without rewriting infrastructure; running GRPO training on a math dataset with a custom reward; wanting PyTorch-native RL tooling instead of a Ray-based stack; scaling an RL experiment with Monarch and TorchTitan.

How do I install torchforge RL Training in Claude Code?

Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill torchforge-rl-training -a claude-code`. Or copy the skill folder (06-post-training/torchforge in Orchestra-Research/AI-Research-SKILLs) into .claude/skills/torchforge-rl-training in your project. Claude Code loads it when a task matches its description.

How do I install torchforge RL Training in Codex?

Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill torchforge-rl-training -a codex`. Or copy the skill folder (06-post-training/torchforge in Orchestra-Research/AI-Research-SKILLs) into .agents/skills/torchforge-rl-training in your project. Codex loads it when a task matches its description.

Can I use torchforge RL Training in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill torchforge-rl-training -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/torchforge-rl-training, .gemini/skills/torchforge-rl-training, .github/skills/torchforge-rl-training and .opencode/skills/torchforge-rl-training in your project.

What does torchforge RL Training need to run?

Going by SKILL.md and its folder, torchforge RL Training needs the command-line tools its instructions call (python and conda). Our summary lists: A conda environment with torchforge installed; Three or more GPUs for GRPO training; A model from the Hugging Face Hub and a training dataset.

Does torchforge RL Training access the network?

SKILL.md names 3 domains. As links in the text: github.com, meta-pytorch.org and discord.gg. This is read from the text; nothing was executed.

Is torchforge RL Training safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does torchforge RL Training use?

torchforge RL Training is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does torchforge RL Training use?

About 2.5k tokens (SKILL.md is roughly 10k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 3.8k tokens, read only when the agent opens those files.

What are the alternatives to torchforge RL Training?

Skills that share tags, products or a category with torchforge RL Training: Magpie Kernel Evaluator (amd/skills, 406 stars), ML Experiment Iteration (Leeroo-AI/superml, 195 stars), Alphagenome Predictions (genomicsxai/alphagenome-pytorch, 162 stars) and ML Training Run Verifier (Leeroo-AI/superml, 195 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains torchforge RL Training?

Orchestra-Research (a GitHub organization) maintains it in Orchestra-Research/AI-Research-SKILLs, which has 13,374 GitHub stars. The repository holds 96 skills in this directory. The repository was last updated on June 16, 2026.

Source: Orchestra-Research/AI-Research-SKILLs on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.