Open Whisper-style Speech Models (OWSM)

Overview

Open Whisper-style Speech Models (OWSM, pronounced as “awesome”) are a series of speech foundation models developed by WAVLab at Carnegie Mellon University. We reproduce Whisper-style training using publicly available data and our open-source toolkit ESPnet. By publicly releasing data preparation scripts, training and inference code, pre-trained model weights and training logs, we aim to promote transparency and open science in large-scale speech pre-training.

Demo

• Gradio demo:
• Colab notebook:

Pre-trained models

We publicly release a series of pre-trained models. The training logs are also available for major models. We recommend using OWSM v3.1 or later versions for better performance and efficiency.

Data details

The latest OWSM v3.1 models are trained on a diverse combination of public datasets as listed below.

The low-restriction model is trained on a subset of the above data with “more flexible licenses”.

Inference

Similar to other ESPnet models, the pre-trained OWSM models can be easily downloaded and used in a python script. Below are some examples using OWSM v3.1. For earlier versions (v2 and before), the language code should follow the two-letter format (e.g., <en>, <de>).

Language Identification

We pass the Hugging Face model tag when initializing Speech2Language. The model will be automatically downloaded from Hugging Face to a local cache directory.

from espnet2.bin.s2t_inference_language import Speech2Language
s2l = Speech2Language.from_pretrained(
    model_tag="espnet/owsm_v3.1_ebf",
    device="cuda",
    nbest=3,  # return nbest prediction and probability
)

import soundfile as sf
speech, rate = sf.read("audio.wav")

result = s2l(speech)
print(result)
# list of tuples (language, probability)
# [('<eng>', 0.9994348883628845), ('<jpn>', 0.00010286537144565955), ('<rus>', 6.185896199895069e-05)]

Speech Recognition or Translation

We use Speech2Text for speech recognition or translation. We also pass the model tag so that the model can be automatically downloaded. When initializing this object, we set the default values for lang_sym, task_sym and predict_time. These variables can be overwritten later, which provides more flexibility. Note that the language must be known to use this functionality. If it is unknown, one can first perform language identification and then recognition or translation.

from espnet2.bin.s2t_inference import Speech2Text
s2t = Speech2Text.from_pretrained(
    model_tag="espnet/owsm_v3.1_ebf",
    device="cuda",
    beam_size=5,
    ctc_weight=0.0,
    maxlenratio=0.0,
    # below are default values which can be overwritten in __call__
    lang_sym="<eng>",
    task_sym="<asr>",
    predict_time=False,
)

import soundfile as sf
speech, rate = sf.read("audio.wav")


result = s2t(speech)[0][-2]

# an optional text prompt can be passed
result = s2t(
    speech,
    text_prev="this is an optional prompt"
)[0][-2]

# lang_sym, task_sym, predict_time can be overwritten
result = s2t(
    speech,
    lang_sym="<eng>",
    task_sym="<st_zho>",    # translation into Chinese
    predict_time=True,
)[0][-2]

Long-form Speech Recognition or Translation

OWSM processes an entire audio recording in a chunk-by-chunk manner. Each chunk has a fixed length of 30s. The chunk is shifted based on the predicted timestamps. We still use Speech2Text but we call its decode_long method.

from espnet2.bin.s2t_inference import Speech2Text
s2t = Speech2Text.from_pretrained(
    model_tag="espnet/owsm_v3.1_ebf",
    device="cuda",
    beam_size=5,
    ctc_weight=0.0,
    maxlenratio=0.0,
    # below are default values which can be overwritten in __call__
    lang_sym="<eng>",
    task_sym="<asr>",
)

import soundfile as sf
speech, rate = sf.read("covid.wav")

result = s2t.decode_long(speech)
# list of tuples (start_time, end_time, text)

Fine-tuning on custom data

Our latest work (accepted to SLT 2024), “ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration”, will provide an easier way for fine-tuning pre-trained models. We are preparing demos and notebooks. Please stay tuned!

Papers

Please cite our papers if you find OWSM helpful.

• ACL 2024: OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification
• INTERSPEECH 2024: On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models
• INTERSPEECH 2024: OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
• ASRU 2023: Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data

We also collect other papers related to OWSM. Please contact Yifan Peng (yifanpen@andrew.cmu.edu) if you use OWSM in your work and want to include it here.