Distil*

Distil* refers to a collection of example scripts and research projects focused on knowledge distillation within the Hugging Face Transformers ecosystem. These scripts demonstrate how to train a smaller 'student' model to mimic the behavior of a larger 'teacher' model, enabling model compression and faster inference while retaining much of the original accuracy. The repository includes examples for distilling BERT-like models on tasks such as language modeling and classification. Distil* is not a standalone product but rather a set of open-source reference implementations intended for developers and researchers working on efficient NLP models.