GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

GLaM is a mixture-of-experts language model introduced in a research paper by Google. It features 1.2 trillion parameters and employs sparse activation to achieve efficient scaling, outperforming GPT-3 with significantly less computational cost. The model is designed for language understanding and generation tasks.