openai /

whisper-small

OpenAI's 244M-parameter model that turns speech into text and translates it into English.

637 likes
3.8M downloads a month
242M parameters
Some experience
Speech

Open on Hugging Face (opens in a new tab)

Whisper Small is a Transformer encoder-decoder model with 244 million parameters. It sits in the middle of the Whisper family, which runs from tiny (39M) to large (1550M). This is the multilingual version, trained for both speech recognition (text in the same language as the audio) and speech translation (text in a different language). It works on audio up to 30 seconds long, and a chunking method in the Transformers pipeline lets it handle longer recordings.

What sets it apart is that it was trained on a huge, varied set of audio, so the card says it handles accents, background noise and technical language well without fine-tuning. The limits are real, though. It can hallucinate, meaning it may write text that was never spoken, and it can repeat itself. It works unevenly across languages: the card says it is strong in about 10 languages and weaker where there is less training data. The card's own Hindi result on Common Voice 11.0 is a word error rate of 87.3, so do not assume every language works well. It is also not built for real-time transcription out of the box.

What you need to run it

242M parameters. The weights take about 483 MB at full 16-bit precision (fits a 8 GB Mac or a GPU with 8 GB), or about 645 MB as a 4-bit download, which runs on a 8 GB Mac or a GPU with 8 GB. Long inputs and images need more memory on top.

What it does well

Many languages, two tasks
The card lists around 100 languages. It can transcribe in the audio's own language or translate speech into English by switching a task token.
Good English accuracy
On LibriSpeech test-clean it scores a word error rate of 3.43, and 7.63 on the harder LibriSpeech test-other set. Lower is better.
Works without fine-tuning
The card says it generalises to many datasets and domains out of the box, with improved robustness to accents, background noise and technical language.
Mid-sized and fine-tunable
At 244M parameters it is much smaller than the large checkpoints. The card points to a guide for fine-tuning Whisper with as little as 5 hours of labelled data.
Long audio and timestamps
Setting chunk_length_s=30 in the pipeline lets it transcribe recordings of any length, and return_timestamps=True adds timestamps.

What you could build

Transcribe English lectures, interviews or podcasts into text
Translate a short clip in French or another supported language into English text
Build a captioning or accessibility tool for recorded audio
Fine-tune it on your own language or accent data and compare it with the original
Measure its word error rate on a dataset to study where it fails

Get started

Install the Transformers library and load the model through the pipeline shown in the card. Setting chunk_length_s=30 lets it handle audio longer than 30 seconds, and the card's example runs on a GPU if one is available or on CPU otherwise.

processor = WhisperProcessor.from_pretrained("openai/whisper-small")

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

Commands are copied from the model card. Read it on Hugging Face for the full setup.