Galactica

Analysis

Why This Paper Matters

Large language models have shown remarkable capabilities across many domains, but their general training often lacks the depth and structure needed for scientific reasoning. Galactica addresses this gap by building a model that is not just exposed to scientific text, but is explicitly designed to handle the unique modalities of science—from chemical formulas to protein sequences to mathematical equations. This matters because scientific discovery increasingly relies on processing vast, heterogeneous datasets, and a model that can seamlessly navigate between natural language and structured scientific data could accelerate research in fields like drug discovery, materials science, and biology.

How It Works

Galactica is a decoder-only Transformer trained on a corpus of 106 billion tokens drawn from papers, reference materials, encyclopedias, and other scientific sources. The dataset includes both natural language and structured sequences like protein sequences and chemical formulas. A key innovation is the specialized tokenization scheme:

• Citations are wrapped with [START_REF] and [END_REF] tokens.
• Step-by-step reasoning is enclosed in <work>...</work> tags, which act as an internal working memory.
• Mathematical content is tokenized by splitting ASCII operations into individual characters and treating parentheses as digits.
• Numbers are split into individual digits.
• SMILES formulas, amino acid sequences, and DNA sequences are each wrapped with their own start/end tokens and tokenized character-by-character.

The <work> token is particularly notable. Transformer models struggle with multi-step reasoning because they lack a persistent working memory. By embedding reasoning steps within <work> tags during training, Galactica learns to use its own output context as a scratchpad. This approach avoids the need for external prompting tricks and provides a bridge toward more robust reasoning capabilities.

Another methodological contribution is prompt pre-training. Rather than relying solely on instruction tuning after training, Galactica includes task prompts directly in the pre-training data. This gives the model a stronger signal for tasks of interest without sacrificing generality. The approach is similar to ExT5 but is applied at scale.

Results

Galactica is evaluated across a wide range of scientific tasks. On a curated set of 434 popular LaTeX equations, it shows strong zero-shot memorization, correctly recalling equations from chemistry, physics, and mathematics.

On domain probes like AminoProbe (amino acid properties), BioLAMA (biomedical facts), and chemical reactions, Galactica consistently outperforms general models. In mathematical reasoning, it exceeds PaLM’s performance with 18 times fewer parameters, especially when using the <work> token.

For downstream scientific NLP tasks like PubMedQA and MedMCQA, Galactica achieves new state-of-the-art results. It also excels at citation prediction, outperforming both sparse and dense retrieval baselines.

In the chemical domain, Galactica predicts IUPAC names from SMILES formulas with increasing accuracy as model size grows, and attention visualization shows it focuses on the correct chemical groups. On MoleculeNet benchmarks, it learns drug discovery tasks from natural language prompts, effectively performing weak supervision.

For protein sequences, Galactica achieves low perplexity on holdout sets and can infer functional keywords from sequences alone, though performance saturates for sequences highly similar to training data.

Significance

Galactica demonstrates that a domain-specific LLM can outperform much larger general models on scientific tasks. Its specialized tokenization and working memory token are practical innovations that could be adopted by other models. The prompt pre-training technique offers a way to improve task performance without sacrificing generality, which is a common trade-off in instruction tuning. By showing that drug discovery tasks can be learned through natural language prompts, Galactica points toward a future where AI can assist in scientific research by directly reading and reasoning over the scientific literature. The main limitation is its bias toward graduate-level knowledge, which causes weaker performance on high-school-level tasks. Future work could address this by including more diverse educational sources and reducing overfitting on highly similar protein sequences.