DBRX by Databricks

DBRX is an open, general-purpose large language model (LLM) developed by Databricks, built on a fine-grained mixture-of-experts (MoE) architecture. With 132 billion total parameters (36 billion active per input), it selects 4 out of 16 experts, enabling superior quality and efficiency. DBRX sets a new state-of-the-art among open LLMs, surpassing GPT-3.5 on standard benchmarks and competing with Gemini 1.0 Pro. It excels at code generation, outperforming specialized models like CodeLLaMA-70B, and also demonstrates strong performance on language understanding (MMLU), math (GSM8K), and RAG tasks. DBRX achieves up to 2× faster inference than LLaMA2-70B and supports up to 150 tokens/second/user when hosted on Databricks Model Serving. Pretrained on 12 trillion tokens of text and code with a maximum context length of 32,000 tokens, the model uses rotary position encodings (RoPE), gated linear units (GLU), grouped query attention (GQA), and the GPT-4 tokenizer. The weights for DBRX Base and DBRX Instruct are available on Hugging Face under an open license, and Databricks customers can access the model via APIs or train custom MoE models using Databricks' training stack.