OpenAI Whisper Reference

OpenAI Whisper is a multilingual speech recognition model that transcribes audio to text in 99 languages and translates non-English audio to English. It is suitable for transcribing podcasts, videos, meeting notes, noisy audio, and multilingual content.

Setup

Prerequisites

Whisper requires Python 3.8-3.11 and ffmpeg installed on your system.

# Requires Python 3.8-3.11
pip install -U openai-whisper

# Requires ffmpeg
# macOS: brew install ffmpeg
# Ubuntu: sudo apt install ffmpeg
# Windows: choco install ffmpeg

For GPU acceleration, you need a CUDA-compatible GPU and drivers. For real-time streaming, install pip install faster-whisper. For LangChain integration, install pip install langchain langchain-chroma langchain-openai.

Model Sizes

Whisper offers several model sizes ranging from 39M to 1550M parameters. VRAM usage ranges from approximately 1 GB to 10 GB. The turbo model has 809M parameters and is recommended for the best speed/quality balance. The base model is recommended for prototyping. The large and turbo models are multilingual only (not English-only).

# Available models
models = ["tiny", "base", "small", "medium", "large", "turbo"]

# Load specific model
model = whisper.load_model("turbo")  # Fastest, good quality

Transcription

Basic Transcription

Load a model and transcribe an audio file. The result contains the full text and segments with timestamps.

import whisper

# Load model
model = whisper.load_model("base")

# Transcribe
result = model.transcribe("audio.mp3")

# Print text
print(result["text"])

# Access segments
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")

Specifying Language

By default, Whisper auto-detects the language. Specifying the language is faster and more accurate.

# Auto-detect language
result = model.transcribe("audio.mp3")

# Specify language (faster)
result = model.transcribe("audio.mp3", language="en")

# Supported: en, es, fr, de, it, pt, ru, ja, ko, zh, and 89 more

Translation to English

Use task="translate" to translate non-English audio to English text.

# Transcription (default)
result = model.transcribe("audio.mp3", task="transcribe")

# Translation to English
result = model.transcribe("spanish.mp3", task="translate")
# Input: Spanish audio → Output: English text

Initial Prompt

Provide context to improve accuracy for technical terms, proper nouns, and domain-specific vocabulary.

# Improve accuracy with context
result = model.transcribe(
    "audio.mp3",
    initial_prompt="This is a technical podcast about machine learning and AI."
)

# Helps with:
# - Technical terms
# - Proper nouns
# - Domain-specific vocabulary

Word-Level Timestamps

Enable word-level timestamps to get start and end times for each word.

# Word-level timestamps
result = model.transcribe("audio.mp3", word_timestamps=True)

for segment in result["segments"]:
    for word in segment["words"]:
        print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")

Temperature Fallback

If confidence is low, Whisper retries with different temperatures to improve results.

# Retry with different temperatures if confidence low
result = model.transcribe(
    "audio.mp3",
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
)

Command Line Usage

Whisper can be used directly from the command line with various options.

# Basic transcription
whisper audio.mp3

# Specify model
whisper audio.mp3 --model turbo

# Output formats
whisper audio.mp3 --output_format txt     # Plain text
whisper audio.mp3 --output_format srt     # Subtitles
whisper audio.mp3 --output_format vtt     # WebVTT
whisper audio.mp3 --output_format json    # JSON with timestamps

# Language
whisper audio.mp3 --language Spanish

# Translation
whisper spanish.mp3 --task translate

Advanced Usage

Batch Processing

Process multiple audio files by looping over a list.

import os

audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]

for audio_file in audio_files:
    print(f"Transcribing {audio_file}...")
    result = model.transcribe(audio_file)

    # Save to file
    output_file = audio_file.replace(".mp3", ".txt")
    with open(output_file, "w") as f:
        f.write(result["text"])

Real-Time Streaming with faster-whisper

For streaming audio, use the faster-whisper library.

# For streaming audio, use faster-whisper
# pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="float16")

# Transcribe with streaming
segments, info = model.transcribe("audio.mp3", beam_size=5)

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

GPU Acceleration

GPU acceleration provides 10-20x speedup over CPU. It is used automatically if available.

import whisper

# Automatically uses GPU if available
model = whisper.load_model("turbo")

# Force CPU
model = whisper.load_model("turbo", device="cpu")

# Force GPU
model = whisper.load_model("turbo", device="cuda")

# 10-20× faster on GPU

Subtitle Generation

Generate SRT subtitles directly from video files.

# Generate SRT subtitles
whisper video.mp4 --output_format srt --language English

# Output: video.srt

Integration with LangChain

Use WhisperTranscriptionLoader to load transcriptions into LangChain for RAG pipelines.

from langchain.document_loaders import WhisperTranscriptionLoader

loader = WhisperTranscriptionLoader(file_path="audio.mp3")
docs = loader.load()

# Use transcription in RAG
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings

vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())

Extract Audio from Video

Use ffmpeg to extract audio from a video file before transcribing.

# Use ffmpeg to extract audio
ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav

# Then transcribe
whisper audio.wav

Performance Notes

Real-time factors on CPU: tiny ~0.32, base ~0.16, turbo ~0.08, large ~1.0. All models achieve ~0.01-0.05 on GPU. A real-time factor of 0.1 means 10x faster than real-time. Whisper has over 72,900 GitHub stars and was trained on 680,000 hours of audio. It is released under the MIT License.

Constraints and Caveats

• Hallucinations: may repeat or invent text.
• Long-form accuracy degrades on audio longer than 30 minutes.
• No speaker identification (diarization).
• Quality varies with accents.
• Background noise can reduce accuracy.
• Not suitable for live captioning due to latency.
• Top-supported languages: en, es, fr, de, it, pt, ru, ja, ko, zh. Quality varies by language.
• Alternatives: AssemblyAI (managed API, speaker diarization), Deepgram (real-time streaming ASR), Google Speech-to-Text (cloud-based).