Whisper Speech Recognition

What it does

When your agent uses it

• Transcribing a podcast or recorded meeting to text
• Translating non-English speech into English text
• Getting word-level timestamps for subtitles
• Batch transcribing a folder of audio files

Example prompts

• “Transcribe ./audio/standup.mp3 with Whisper and save the text.”
• “Translate this Spanish interview recording into English using the turbo model.”
• “Transcribe every mp3 in ./podcast/episodes and include word timestamps.”

Requirements

• Python 3.8 to 3.11 with the openai-whisper package
• Enough VRAM for the model size, from about 1 GB for tiny to about 10 GB for large

Workflow steps

10 steps, taken from the first numbered list in SKILL.md.

1. 1Use turbo model - Best speed/quality for English
2. 2Specify language - Faster than auto-detect
3. 3Add initial prompt - Improves technical terms
4. 4Use GPU - 10-20× faster
5. 5Batch process - More efficient
6. 6Convert to WAV - Better compatibility
7. 7Split long audio - <30 min chunks
8. 8Check language support - Quality varies by language
9. 9Use faster-whisper - 4× faster than openai-whisper
10. 10Monitor VRAM - Scale model size to hardware

What it can do on your machine

Read from SKILL.md and the folder at commit 773a529. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • whisper
  • pip
  • ffmpeg

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Links to these hosts (documentation or services it may open):

  • github.com
  • arxiv.org

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Whisper Speech Recognition loads about 1.9k tokens when it runs, and up to ~3.1k if it reads all its reference files. Until then it costs about 82 tokens; SKILL.md has 326 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~82

When it runs · the whole SKILL.md, loaded when a task matches

~1.9k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~3.1k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NoteRuns commands with sudoSKILL.md:46

  # Ubuntu: sudo apt install ffmpeg

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from Orchestra-Research/AI-Research-SKILLs at commit 773a529, republished under its MIT licence (© Orchestra-Research). 326 words, ~1,859 tokens.

name

whisper

description

OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.

version

1.0.0

author

Orchestra Research

license

MIT

tags

Whisper, Speech Recognition, ASR, Multimodal, Multilingual, OpenAI, Speech-To-Text, Transcription, Translation, Audio Processing

dependencies

openai-whisper, transformers, torch

# Requires Python 3.8-3.11
pip install -U openai-whisper

# Requires ffmpeg
# macOS: brew install ffmpeg
# Ubuntu: sudo apt install ffmpeg
# Windows: choco install ffmpeg

import whisper

# Load model
model = whisper.load_model("base")

# Transcribe
result = model.transcribe("audio.mp3")

# Print text
print(result["text"])

# Access segments
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")

# Available models
models = ["tiny", "base", "small", "medium", "large", "turbo"]

# Load specific model
model = whisper.load_model("turbo")  # Fastest, good quality

# Auto-detect language
result = model.transcribe("audio.mp3")

# Specify language (faster)
result = model.transcribe("audio.mp3", language="en")

# Supported: en, es, fr, de, it, pt, ru, ja, ko, zh, and 89 more

# Transcription (default)
result = model.transcribe("audio.mp3", task="transcribe")

# Translation to English
result = model.transcribe("spanish.mp3", task="translate")
# Input: Spanish audio → Output: English text

# Improve accuracy with context
result = model.transcribe(
    "audio.mp3",
    initial_prompt="This is a technical podcast about machine learning and AI."
)

# Helps with:
# - Technical terms
# - Proper nouns
# - Domain-specific vocabulary

# Word-level timestamps
result = model.transcribe("audio.mp3", word_timestamps=True)

for segment in result["segments"]:
    for word in segment["words"]:
        print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")

# Retry with different temperatures if confidence low
result = model.transcribe(
    "audio.mp3",
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
)

# Basic transcription
whisper audio.mp3

# Specify model
whisper audio.mp3 --model turbo

# Output formats
whisper audio.mp3 --output_format txt     # Plain text
whisper audio.mp3 --output_format srt     # Subtitles
whisper audio.mp3 --output_format vtt     # WebVTT
whisper audio.mp3 --output_format json    # JSON with timestamps

# Language
whisper audio.mp3 --language Spanish

# Translation
whisper spanish.mp3 --task translate

import os

audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]

for audio_file in audio_files:
    print(f"Transcribing {audio_file}...")
    result = model.transcribe(audio_file)

    # Save to file
    output_file = audio_file.replace(".mp3", ".txt")
    with open(output_file, "w") as f:
        f.write(result["text"])

# For streaming audio, use faster-whisper
# pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="float16")

# Transcribe with streaming
segments, info = model.transcribe("audio.mp3", beam_size=5)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

import whisper

# Automatically uses GPU if available
model = whisper.load_model("turbo")

# Force CPU
model = whisper.load_model("turbo", device="cpu")

# Force GPU
model = whisper.load_model("turbo", device="cuda")

# 10-20× faster on GPU

# Generate SRT subtitles
whisper video.mp4 --output_format srt --language English

# Output: video.srt

from langchain.document_loaders import WhisperTranscriptionLoader

loader = WhisperTranscriptionLoader(file_path="audio.mp3")
docs = loader.load()

# Use transcription in RAG
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

# Use ffmpeg to extract audio
ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav

# Then transcribe
whisper audio.wav

© Orchestra-Research, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file (references) in 18-multimodal/whisper of Orchestra-Research/AI-Research-SKILLs.

• SKILL.md
• references/languages.md

Open the folder on GitHubat commit 773a529

Used in 7 other repositories

We found 7 copies of this SKILL.md (exact, near-identical or edited) in other folders, from 7 other GitHub owners. This page covers the copy in Orchestra-Research/AI-Research-SKILLs, which our catalogue first saw on October 7, 2026.

• RedWoodOG/Hermes-Desktopskills/mlops/models/whisper
• Tommy-yw/RunbookHermesoptional-skills/mlops/whisper
• Luciole-Studio/Misaka-Agentmisaka/core/skills/assets/optional/mlops/whisper
• davila7/claude-code-templatescli-tool/components/skills/ai-research/multimodal-whisper
• NousResearch/hermes-agentoptional-skills/mlops/whisper
• johnson7788/MultiUserClawhermes-agent/optional-skills/mlops/whisper
• OpenLAIR/dr-clawskills/multimodal/whisper