Whisper

Whisper is a general-purpose speech recognition model used for various applications. It's trained on a large, diverse audio dataset, enabling it to handle tasks like multilingual speech recognition, speech translation, and language identification.

Whisper is an automatic speech recognition (ASR) system, using a training dataset of 680,000 hours of multitask, multilingual data from the internet. This dataset helps it handle accents, background noise, and technical language. It also supports transcription in multiple languages and translation into English. Its models and inference code are open-source, aiding application development and further research.

Whisper uses an end-to-end approach through an encoder-decoder Transformer. It processes 30-second audio segments into a log-Mel spectrogram, which is fed into an encoder. A decoder predicts the corresponding text caption and special tokens for tasks like language identification and translation into English. About a third of Whisper’s audio dataset is non-English, and it alternates between transcribing in the original language or translating to English. This method has outperformed the supervised state-of-the-art on CoVoST2 to English translation in a zero-shot scenario.