Neura Market
MPT-7B logo

MPT-7B

Free

A New Standard for Open-Source, Commercially Usable LLMs

FreeFree tier
Inputs: textOutputs: text
Type
Open Source
Founded
2013
Company
Databricks
Visit Website GitHub (38 stars)

About MPT-7B

MPT-7B (MosaicML Pretrained Transformer 7B) is an open-source, commercially usable large language model released by MosaicML (a Databricks company). It is a decoder-style transformer with 6.7 billion parameters, trained from scratch on 1 trillion tokens of text and code. The model achieves quality comparable to LLaMA-7B while being fully open source under the Apache 2.0 license (base model). Key technical innovations include ALiBi (Attention with Linear Biases) enabling extrapolation to context lengths up to 84k tokens, FlashAttention for efficient training and inference, and integration with FasterTransformer. MPT-7B was trained on the MosaicML platform in 9.5 days at a cost of approximately $200,000, with zero human intervention. Alongside the base model, three finetuned variants are released: MPT-7B-StoryWriter-65k+ (for long-form story generation with 65k+ token context), MPT-7B-Instruct (for short-form instruction following), and MPT-7B-Chat (for dialogue generation). The entire training, finetuning, and evaluation codebase is open-sourced via the MosaicML LLM Foundry.

Key Features

Commercially usable open-source license (Apache 2.0 for base model)
Trained on 1 trillion tokens of text and code
ALiBi for long context extrapolation up to 84k tokens
FlashAttention and FasterTransformer for fast training and inference
Matches or surpasses quality of LLaMA-7B
Three finetuned variants: StoryWriter-65k+, Instruct, Chat
Open-source training code via MosaicML LLM Foundry
Trained on MosaicML platform in 9.5 days with zero human intervention at ~$200k cost

Pros & Cons

Pros
  • Open source and available for commercial use (base model Apache 2.0)
  • Very long context window (up to 84k tokens) enabling document-level tasks
  • High quality comparable to LLaMA-7B
  • Efficient training and inference with FlashAttention
  • Multiple finetuned variants for different use cases
  • Full open-source codebase for reproducibility and customization
Cons
  • Smaller model size (7B parameters) may limit performance on complex reasoning
  • Some finetuned variants (Chat) use non-commercial licenses
  • Training from scratch requires significant compute resources (~$200k)
  • Base model may require additional fine-tuning for production-quality instruction following

Best For

Long-form story and fiction writing (65k+ token context)Short-form instruction following and question answeringDialogue generation and chatbot applicationsGeneral text and code generation tasksFine-tuning on proprietary datasets for domain-specific applications

FAQ

What is MPT-7B?
MPT-7B is a 6.7 billion parameter open-source transformer language model trained on 1 trillion tokens of text and code, released by MosaicML (Databricks). It is licensed under Apache 2.0 for commercial use and matches the quality of LLaMA-7B.
Is MPT-7B free for commercial use?
Yes, the base MPT-7B model is released under the Apache 2.0 license, which allows commercial use. However, the MPT-7B-Chat variant uses a CC-By-NC-SA-4.0 license (non-commercial only).
What context length does MPT-7B support?
Due to ALiBi (Attention with Linear Biases), MPT-7B was trained on up to 65k token inputs and can extrapolate to at least 84k tokens at inference time, far exceeding typical 2k-4k limits of other open-source models.
How was MPT-7B trained?
MPT-7B was trained on the MosaicML platform in 9.5 days with zero human intervention, costing approximately $200,000. It used 1 trillion tokens of text and code curated by MosaicML's data team.
What are the different versions of MPT-7B?
There are four versions: MPT-7B Base (Apache 2.0), MPT-7B-StoryWriter-65k+ (for long story generation), MPT-7B-Instruct (for instruction following), and MPT-7B-Chat (for dialogue generation, non-commercial).