Fine-Tuning Expert

What it does

When your agent uses it

• Adapting a foundation model to a specific task with LoRA or QLoRA adapters
• Preparing and checking a JSONL training dataset before a run
• Choosing learning rates, batch sizes and schedulers for a fine-tuning job
• Comparing a tuned model with its base model on held-out data
• Merging adapter weights and quantizing a tuned model for serving

Example prompts

• “Set up a QLoRA run on our support transcripts that fits on one consumer GPU.”
• “Check data.jsonl for formatting errors before I start training.”
• “Our validation loss rises after the second epoch, so help me work out whether it is overfitting.”
• “Merge the LoRA adapter into the base model and measure inference throughput.”

Requirements

• Python with Hugging Face `transformers`, `peft` and `datasets`

Workflow steps

5 steps, taken from the first numbered list in SKILL.md.

1. 1Dataset preparation — Validate and format data; run quality checks before training starts
2. 2Method selection — Choose PEFT technique based on GPU memory and task requirements
3. 3Training — Configure hyperparameters, monitor loss curves, checkpoint regularly
4. 4Evaluation — Benchmark against the base model; test on held-out set and edge cases
5. 5Deployment — Merge adapter weights, quantize, measure inference throughput before serving

What it can do on your machine

Read from SKILL.md and the folder at commit 1be15d8. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • python

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • github.com
  • synergetic.solutions
  • jeffallan.github.io

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Fine-Tuning Expert loads about 1.7k tokens when it runs, and up to ~22k if it reads all its reference files. Until then it costs about 135 tokens; SKILL.md has 333 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~135

When it runs · the whole SKILL.md, loaded when a task matches

~1.7k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~22k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Jeffallan/claude-skills at commit 1be15d8, republished under its MIT licence (© Jeffallan). 333 words, ~1,695 tokens.

name

fine-tuning-expert

description

Use when fine-tuning LLMs, training custom models, or adapting foundation models for specific tasks. Invoke for configuring LoRA/QLoRA adapters, preparing JSONL training datasets, setting hyperparameters for fine-tuning runs, adapter training, transfer learning, finetuning with Hugging Face PEFT, OpenAI fine-tuning, instruction tuning, RLHF, DPO, or quantizing and deploying fine-tuned models. Trigger terms include: LoRA, QLoRA, PEFT, finetuning, fine-tuning, adapter tuning, LLM training, model training, custom model.

license

MIT

metadata.author

https://github.com/Jeffallan

metadata.company

https://synergetic.solutions

metadata.version

1.1.0

metadata.domain

data-ml

metadata.triggers

fine-tuning, fine tuning, finetuning, LoRA, QLoRA, PEFT, adapter tuning, transfer learning, model training, custom model, LLM training, instruction tuning…

metadata.role

expert

metadata.scope

implementation

metadata.output-format

code

metadata.related-skills

devops-engineer

from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer
import torch

# 1. Load base model and tokenizer
model_id = "meta-llama/Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# 2. Configure LoRA adapter
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,               # rank — increase for more capacity, decrease to save memory
    lora_alpha=32,      # scaling factor; typically 2× rank
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # verify: should be ~0.1–1% of total params

# 3. Load and format dataset (Alpaca-style JSONL)
dataset = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

def format_prompt(example):
    return {"text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"}

dataset = dataset.map(format_prompt)

# 4. Training arguments
training_args = TrainingArguments(
    output_dir="./checkpoints",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,     # effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,                 # always use warmup
    fp16=False,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_steps=200,
    load_best_model_at_end=True,
)

# 5. Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    dataset_text_field="text",
    max_seq_length=2048,
)
trainer.train()

# 6. Save adapter weights only
model.save_pretrained("./lora-adapter")
tokenizer.save_pretrained("./lora-adapter")

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device_map="auto")

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, "./lora-adapter").merge_and_unload()
merged.save_pretrained("./merged-model")

© Jeffallan, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 5 other files (references) in skills/fine-tuning-expert of Jeffallan/claude-skills.

• SKILL.md
• references/dataset-preparation.md
• references/deployment-optimization.md
• references/evaluation-metrics.md
• references/hyperparameter-tuning.md
• references/lora-peft.md

Open the folder on GitHubat commit 1be15d8