New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Dl Transformer Finetune

by wentorai in wentorai/research-plugins

Build transformer fine-tuning plans for classification and generation

MITAuto-check passedAI & LLM Engineering

Install Dl Transformer Finetune

skills CLI
$ npx skills add wentorai/research-plugins --skill dl-transformer-finetune -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install wentorai/research-plugins dl-transformer-finetune --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/wentorai/research-plugins.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/domains/ai-ml/dl-transformer-finetune .claude/skills/dl-transformer-finetune && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
dl-transformer-finetune
GitHub stars
298
Used in
1 other repo
Token cost
~2.2k tokens
SKILL.md length
422 words
Files
1
Skills in repo
405
Repo updated
First seen
Licence
MIT

At a glance

Build transformer fine-tuning plans for classification and generation

  • Tasks that involve Fine-tuning
  • SKILL.md covers Overview, Full Fine-Tuning, Parameter-Efficient… and PEFT Method Comparison, plus 4 more sections
  • Instructions only: no scripts, shell commands, URLs or credentials in SKILL.md

What it does

Dl Transformer Finetune is an agent skill from wentorai/research-plugins. Build transformer fine-tuning plans for classification and generation

Its SKILL.md is about 2.2k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.

It sits in AI & LLM Engineering, covering Fine-tuning. The repository describes itself as: 350+ academic research skills, MCP configs, and plugins for Research-Claw and AI agents. The licence is MIT.

When your agent uses it

  • Tasks that involve Fine-tuning

Example prompts

  • “/dl-transformer-finetune”

Requirements

  • Python 3

What it can do on your machine

Read from SKILL.md and the folder at commit bf44b3c. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    No scripts in the folder and no shell commands in SKILL.md (its code samples are python).

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    Links to these hosts (documentation or services it may open):

    • arxiv.org
    • github.com
    • huggingface.co

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names no API keys, tokens, secrets or passwords.

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Dl Transformer Finetune loads about 2.2k tokens when it runs. Until then it costs about 23 tokens; SKILL.md has 422 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~23
When it runs · the whole SKILL.md, loaded when a task matches
~2.2k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from wentorai/research-plugins at commit bf44b3c, republished under its MIT licence (© wentorai). 422 words, ~2,178 tokens.

Download SKILL.mdSave it as .claude/skills/dl-transformer-finetune/SKILL.md (or your agent's skills folder).
name
dl-transformer-finetune
description
Build transformer fine-tuning plans for classification and generation

Transformer Fine-Tuning Guide

Overview

Fine-tuning pretrained transformers is the dominant paradigm in modern NLP and increasingly in vision, audio, and multimodal research. The core idea is simple: take a model pretrained on massive data, then adapt it to your specific task with a comparatively small labeled dataset. But the practical details -- which layers to freeze, which optimizer and learning rate to use, how to handle catastrophic forgetting, when to use parameter-efficient methods -- determine whether fine-tuning succeeds or fails.

This guide covers the full spectrum of fine-tuning approaches: full fine-tuning for maximum performance, parameter-efficient fine-tuning (PEFT) for resource-constrained settings, and the decision framework for choosing between them. The patterns are drawn from hundreds of published papers and the Hugging Face ecosystem that supports them.

Whether you are fine-tuning BERT for text classification in a domain-specific corpus, adapting a large language model with LoRA for instruction following, or building a multi-task model for your research pipeline, this guide provides the recipes you need.

Full Fine-Tuning

Text Classification with BERT
python
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
)
from datasets import load_dataset
import numpy as np
from sklearn.metrics import accuracy_score, f1_score

# Load model and tokenizer
model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=3
)

# Prepare dataset
dataset = load_dataset("multi_nli")

def tokenize_function(examples):
    return tokenizer(
        examples["premise"],
        examples["hypothesis"],
        truncation=True,
        max_length=128,
        padding="max_length",
    )

tokenized = dataset.map(tokenize_function, batched=True)

# Metrics
def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    return {
        "accuracy": accuracy_score(labels, preds),
        "f1_macro": f1_score(labels, preds, average="macro"),
    }

# Training arguments (research-grade defaults)
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    per_device_eval_batch_size=64,
    learning_rate=2e-5,                  # Standard for BERT fine-tuning
    weight_decay=0.01,
    warmup_ratio=0.06,                   # 6% warmup
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="f1_macro",
    fp16=True,
    dataloader_num_workers=4,
    seed=42,
    report_to="wandb",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["validation_matched"],
    compute_metrics=compute_metrics,
)

trainer.train()
Learning Rate Selection Guide
Model SizeRecommended LRWarmupWeight Decay
BERT-base (110M)2e-5 to 5e-56-10%0.01
BERT-large (340M)1e-5 to 3e-56-10%0.01
RoBERTa-large (355M)1e-5 to 2e-56%0.01
T5-base (220M)3e-4 to 1e-30-5%0.01
LLaMA-7B (full FT)1e-5 to 2e-53%0.0
LLaMA-7B (LoRA)1e-4 to 3e-43%0.0

Parameter-Efficient Fine-Tuning (PEFT)

LoRA (Low-Rank Adaptation)

LoRA freezes the pretrained weights and injects trainable low-rank decomposition matrices. It typically trains only 0.1-1% of parameters while achieving 95-100% of full fine-tuning performance.

python
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Configure LoRA
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                          # Rank (8-64 typical)
    lora_alpha=32,                 # Scaling factor (usually 2*r)
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.062
Show full SKILL.md (170 more words)Show less
QLoRA (Quantized LoRA)
python
from transformers import BitsAndBytesConfig

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

# Apply LoRA on top of quantized model
model = get_peft_model(model, lora_config)
# Now fits on a single 24GB GPU!

PEFT Method Comparison

MethodTrainable %MemoryPerformanceBest For
Full fine-tuning100%HighBestSufficient compute + data
LoRA0.1-1%Low95-100%Most scenarios
QLoRA0.1-1%Very low93-98%Consumer GPUs
Prefix tuning~0.1%Low90-95%Generation tasks
Adapter layers1-5%Medium95-99%Multi-task
Prompt tuning<0.01%Minimal85-95%Large models, many tasks

Avoiding Catastrophic Forgetting

python
# Strategy 1: Gradual unfreezing (Howard & Ruder, 2018)
def gradual_unfreeze(model, epoch, total_layers=12):
    """Unfreeze one more layer group per epoch, from top to bottom."""
    layers_to_unfreeze = min(epoch + 1, total_layers)
    for i, (name, param) in enumerate(reversed(list(model.named_parameters()))):
        param.requires_grad = i < layers_to_unfreeze * 10  # ~10 params per layer

# Strategy 2: Discriminative learning rates
def get_layer_lrs(model, base_lr=2e-5, decay_factor=0.95):
    """Apply lower learning rates to earlier layers."""
    params = []
    num_layers = 12  # BERT-base
    for i in range(num_layers):
        lr = base_lr * (decay_factor ** (num_layers - i - 1))
        layer_params = [p for n, p in model.named_parameters()
                       if f"layer.{i}." in n]
        params.append({"params": layer_params, "lr": lr})
    return params

# Strategy 3: EWC (Elastic Weight Consolidation)
# Add a penalty term that keeps important weights close to pretrained values

Fine-Tuning Checklist for Papers

Before fine-tuning:
[ ] Report exact pretrained model name and version
[ ] Document dataset size, splits, and preprocessing
[ ] Specify hardware (GPU model, count, precision)
[ ] Set random seeds (Python, NumPy, PyTorch, CUDA)

During fine-tuning:
[ ] Use validation set for hyperparameter selection
[ ] Log training curves (loss, metrics per epoch)
[ ] Monitor for overfitting (val loss divergence)
[ ] Try at least 3 learning rates from the recommended range

Reporting:
[ ] Report mean and std across 3-5 random seeds
[ ] Include training time and compute cost
[ ] Compare against published baselines using same evaluation
[ ] Release model weights or LoRA adapters for reproducibility

Best Practices

  • Start with the recommended learning rate for your model size, then sweep 3-5 values.
  • Use LoRA first unless you have strong evidence that full fine-tuning is needed.
  • Always evaluate on a held-out test set that was not used for any hyperparameter decisions.
  • Freeze embeddings when fine-tuning for classification -- they rarely need updating.
  • Use gradient accumulation to simulate larger batch sizes on limited hardware.
  • Save the tokenizer alongside the model to ensure reproducibility.

References

© wentorai, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

Just SKILL.md in skills/domains/ai-ml/dl-transformer-finetune of wentorai/research-plugins.

Open the folder on GitHubat commit bf44b3c

Used in 1 other repository

We found 1 copy of this SKILL.md (exact, near-identical or edited) in other folders, from 1 other GitHub owner. This page covers the copy in wentorai/research-plugins, which our catalogue first saw on October 7, 2026.

Compare with similar skills

Dl Transformer Finetune next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Dl Transformer Finetune compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Dl Transformer Finetune this skillwentorai/research-plugins2981 repos~2.2kAutomated safety check: PassMIT
Sentence-Transformers Training Routerhuggingface/skills11k1 repos~2.6kAutomated safety check: PassApache-2.0
Train RlOpenPipe/ART11k—~2.4kAutomated safety check: PassApache-2.0
Qwopus27b Rl TrainingR6410418/Jackrong-llm-finetuning-guide1.7k—~830Automated safety check: PassApache-2.0
Dataset Evaluationawslabs/agent-plugins9161 repos~1.3kAutomated safety check: PassApache-2.0
Train SftOpenPipe/ART11k—~2.9kAutomated safety check: PassApache-2.0

Similar skills

  • Sentence-Transformers Training Router

    huggingface/skills

    Official

    Routes a sentence-transformers training task to the right model type and required reference docs and example scripts, covering bi-encoders, rerankers, sparse and multi-vector models.

    11k GitHub starsUsed in 1 repo~2.6k tokens
    AI & LLM EngineeringAuto-check passed

  • Train Rl

    OpenPipe/ART

    RL training reference for the ART framework. An agent skill from OpenPipe/ART.

    11k GitHub stars~2.4k tokensUpdated today
    AI & LLM EngineeringAuto-check passed

  • Qwopus27b Rl Training

    R6410418/Jackrong-llm-finetuning-guide

    Prepare, validate, launch-plan, monitor, resume, and stop configurable Qwopus 27B reinforcement-learning workflows for GRPO or GSPO.

    1.7k GitHub stars~830 tokensUpdated 3 mo ago
    AI & LLM EngineeringAuto-check passed

  • Dataset Evaluation

    awslabs/agent-plugins

    Official

    Validates dataset formatting and quality for SageMaker model fine-tuning (SFT, DPO, or RLVR).

    916 GitHub starsUsed in 1 repo~1.3k tokens
    AI & LLM EngineeringAuto-check passed

  • Train Sft

    OpenPipe/ART

    SFT training reference for the ART framework. An agent skill from OpenPipe/ART.

    11k GitHub stars~2.9k tokensUpdated today
    AI & LLM EngineeringAuto-check passed

  • Fine Tuning With Trl

    Orchestra-Research/AI-Research-SKILLs

    Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training.

    13k GitHub starsUsed in 6 repos~2.9k tokens
    AI & LLM EngineeringAuto-check passed

More from wentorai/research-plugins

All 405 skills in this repo

  • Abstract Writing Guide

    wentorai/research-plugins

    Craft structured research abstracts that maximize clarity and journal acceptance

    298 GitHub starsUsed in 1 repo~1.7k tokens
    Auto-check passed

  • Academic Citation Manager

    wentorai/research-plugins

    Manage academic citations across BibTeX, APA, MLA, and Chicago formats

    298 GitHub starsUsed in 1 repo~2.7k tokens
    Auto-check passed

  • Academic Paper Summarizer

    wentorai/research-plugins

    Summarize academic papers with structured extraction of key elements

    298 GitHub starsUsed in 1 repo~1.4k tokens
    Auto-check passed

  • Academic Study Methods

    wentorai/research-plugins

    Evidence-based study techniques for academic learning and retention

    298 GitHub starsUsed in 1 repo~1.8k tokens
    Auto-check passed

  • Academic Tone Guide

    wentorai/research-plugins

    Adjust writing tone and register for academic audiences and venues

    298 GitHub starsUsed in 1 repo~1.9k tokens
    Auto-check passed

  • Academic Translation Guide

    wentorai/research-plugins

    Academic translation, post-editing, and Chinglish correction guide

    298 GitHub starsUsed in 1 repo~1.6k tokens
    Auto-check passed

Categories

Questions about Dl Transformer Finetune

What does Dl Transformer Finetune do?

Build transformer fine-tuning plans for classification and generation. Dl Transformer Finetune is an agent skill from wentorai/research-plugins.

When should I use Dl Transformer Finetune?

Dl Transformer Finetune fits situations like: tasks that involve Fine-tuning.

How do I install Dl Transformer Finetune in Claude Code?

Run `npx skills add wentorai/research-plugins --skill dl-transformer-finetune -a claude-code`. Or copy the skill folder (skills/domains/ai-ml/dl-transformer-finetune in wentorai/research-plugins) into .claude/skills/dl-transformer-finetune in your project. Claude Code loads it when a task matches its description.

How do I install Dl Transformer Finetune in Codex?

Run `npx skills add wentorai/research-plugins --skill dl-transformer-finetune -a codex`. Or copy the skill folder (skills/domains/ai-ml/dl-transformer-finetune in wentorai/research-plugins) into .agents/skills/dl-transformer-finetune in your project. Codex loads it when a task matches its description.

Can I use Dl Transformer Finetune in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add wentorai/research-plugins --skill dl-transformer-finetune -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/dl-transformer-finetune, .gemini/skills/dl-transformer-finetune, .github/skills/dl-transformer-finetune and .opencode/skills/dl-transformer-finetune in your project.

What does Dl Transformer Finetune need to run?

SKILL.md names no scripts, command-line tools or credentials: Dl Transformer Finetune is instructions for the agent only. Our summary lists: Python 3.

Does Dl Transformer Finetune access the network?

SKILL.md names 3 domains. As links in the text: arxiv.org, github.com and huggingface.co. This is read from the text; nothing was executed.

Is Dl Transformer Finetune safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.

What licence does Dl Transformer Finetune use?

Dl Transformer Finetune is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Dl Transformer Finetune use?

About 2.2k tokens (SKILL.md is roughly 8.7k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.

What are the alternatives to Dl Transformer Finetune?

Skills that share tags, products or a category with Dl Transformer Finetune: Sentence-Transformers Training Router (huggingface/skills, 11k stars), Train Rl (OpenPipe/ART, 11k stars), Qwopus27b Rl Training (R6410418/Jackrong-llm-finetuning-guide, 1.7k stars) and Dataset Evaluation (awslabs/agent-plugins, 916 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Dl Transformer Finetune?

wentorai (a GitHub user) maintains it in wentorai/research-plugins, which has 298 GitHub stars. The repository holds 405 skills in this directory. The repository was last updated on June 19, 2026.

Source: wentorai/research-plugins on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.