HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units

Analysis

Why This Paper Matters

Self-supervised learning has become a cornerstone of modern AI, enabling models to learn from vast unlabeled data. In speech, this is particularly valuable because labeled data is expensive and scarce. HuBERT addresses three unique challenges in speech: multiple sound units per utterance, no lexicon during pre-training, and variable-length units without explicit segmentation. By leveraging offline clustering to create aligned targets for masked prediction, HuBERT provides a simple yet effective framework that rivals wav2vec 2.0, a leading self-supervised speech model. This work matters because it demonstrates that consistent clustering, rather than perfect cluster quality, can drive strong representation learning, opening avenues for simpler and more robust pre-training methods.

Technical Contributions

• Masked Prediction of Hidden Units: HuBERT applies a BERT-like prediction loss over masked regions, using cluster assignments as targets. This forces the model to learn both acoustic and language information from continuous inputs.
• Offline Clustering with k-means: A simple k-means teacher with 100 clusters generates targets. The process is iterated twice, refining the clusters and improving the model.
• Handling Variable-Length Units: By predicting cluster labels for each frame, HuBERT implicitly handles variable-length sound units without requiring explicit segmentation.
• Scalability: The method scales to large datasets (Libri-light 60,000h) and large models (1B parameters), showing consistent gains.

Results

HuBERT matches or improves upon wav2vec 2.0 on Librispeech (960h) and Libri-light (60,000h) across fine-tuning subsets of 10min, 1h, 10h, 100h, and 960h. Notably, with a 1B parameter model, HuBERT achieves up to 19% and 13% relative word error rate (WER) reduction on the challenging dev-other and test-other subsets, respectively. These results highlight the effectiveness of the clustering-based masked prediction approach.

Significance

HuBERT's significance lies in its simplicity and performance. It shows that a straightforward clustering step combined with masked prediction can yield state-of-the-art speech representations, reducing the need for complex architectures or auxiliary losses. This has implications for low-resource languages and domains where labeled data is limited. Moreover, the approach is general and could inspire similar methods in other sequential data modalities. HuBERT has already influenced subsequent research in self-supervised speech and multimodal learning, solidifying its place as a key contribution to the field.