What can you build with whisper?

Transcribe interviews, lectures or podcasts into text files. Translate recordings of non-English speech into English text. Detect which language is spoken in an audio clip. Build a Python app that adds speech-to-text to your own tool. Compare model sizes to find the best speed and accuracy for your hardware.

openai /

whisper

Speech recognition model that transcribes and translates audio in many languages

111.1k stars
1.2k stars since 3 Oct
Python
Updated 2 months ago
Beginner friendly
Image, video & audio

Open on GitHub (opens in a new tab)

Whisper turns spoken audio into text. It is a single Transformer model trained on a large, varied set of audio. One model handles several jobs: multilingual speech recognition, speech translation, language identification and voice activity detection.

It splits audio into 30-second windows and predicts text token by token for each window. Special tokens tell the decoder which task to do, such as transcribe or translate. This lets one model replace several stages of a traditional speech-processing pipeline.

You pick a model size to fit your hardware. Tiny needs about 1 GB of VRAM and runs fast. Large needs about 10 GB and is more accurate. Turbo is an optimized large-v3 that is faster with a small loss in accuracy. Code and weights are under the MIT license.

What it does

Multilingual transcription
Transcribes speech in many languages. You can name the language or let the model detect it. Accuracy varies by language, and the README shows per-language error rates.
Translation into English
With --task translate and a multilingual model such as medium or large, non-English speech comes out as English text. The turbo model is not trained for translation.
Six model sizes
Models range from tiny (39M parameters) to large (1550M). Four sizes have English-only versions that often do better on English.
Command line and Python API
The whisper command transcribes one or more audio files. In Python, load_model and transcribe return the text. detect_language and decode give lower-level control.

What you could build

Transcribe interviews, lectures or podcasts into text files.
Translate recordings of non-English speech into English text.
Detect which language is spoken in an audio clip.
Build a Python app that adds speech-to-text to your own tool.
Compare model sizes to find the best speed and accuracy for your hardware.

Get started

Install the package with pip and install ffmpeg with your system's package manager (the brew command is for macOS). Then run the whisper command on an audio file. You may need Rust if no prebuilt tiktoken package exists for your platform.

pip install -U openai-whisper

brew install ffmpeg

whisper audio.flac audio.mp3 audio.wav --model turbo

Commands are copied from the project's README. Read it on GitHub for the full setup.

Who it's for

Suits developers and students who want speech-to-text and can use a terminal or basic Python; you need ffmpeg installed, and a GPU helps for larger models.

Useful for

AI engineersDrop-in speech-to-text for voice features, with model sizes to trade speed against accuracy.
Creators and marketersTranscribe podcasts, interviews and videos into text for captions, notes and repurposed posts.
ML engineers and data scientistsStudy and benchmark a widely used multilingual speech recognition model across sizes.
Students and beginnersSimple CLI and Python API for learning speech recognition hands-on.