Advancing LLM Reasoning Generalists with Preference Trees

Free

FreeFree tier

Type

Open Source

About Advancing LLM Reasoning Generalists with Preference Trees

Eurus is a suite of large language models (LLMs) optimized for reasoning, fine-tuned from Mistral-7B and CodeLlama-70B. It achieves state-of-the-art results among open-source models on benchmarks covering mathematics, code generation, and logical reasoning. Notably, Eurus-70B surpasses GPT-3.5 Turbo in comprehensive reasoning tests, with a 33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, outperforming existing open-source models by over 13.3%. Its strong performance is attributed to UltraInteract, a newly curated large-scale, high-quality alignment dataset designed for complex reasoning tasks, which includes preference trees with diverse reasoning chains, multi-turn interaction trajectories, and pairwise data for preference learning.

Key Features

State-of-the-art results among open-source models on mathematics, code generation, and logical reasoning benchmarks

Fine-tuned from Mistral-7B and CodeLlama-70B base models

Surpasses GPT-3.5 Turbo across 12 tests covering five reasoning tasks

UltraInteract alignment dataset: large-scale, high-quality, with preference trees for complex reasoning

Novel reward modeling objective derived from preference learning exploration

Outperforms existing open-source models by >13.3% on LeetCode and TheoremQA

Pros & Cons

Pros

Achieves state-of-the-art reasoning performance among open-source models
Surpasses GPT-3.5 Turbo on a comprehensive reasoning benchmark suite
Large margins of improvement on challenging benchmarks like LeetCode and TheoremQA
Open-source models and data are publicly available
UltraInteract dataset enables effective preference learning for reasoning

Cons

Primarily evaluated on reasoning benchmarks; general conversational ability and other NLP tasks not assessed
May require significant computational resources for inference due to model size (70B parameters)
Dependence on base models (Mistral-7B, CodeLlama-70B) which have their own limitations

Best For

Mathematical reasoningCode generation and understandingLogical reasoning and problem-solvingComplex multi-step reasoning tasksBenchmark evaluation for LLM reasoning capabilities

FAQ

What is Eurus?

Eurus is a suite of large language models optimized for reasoning, fine-tuned from Mistral-7B and CodeLlama-70B, achieving state-of-the-art results on mathematics, code generation, and logical reasoning benchmarks.

What is UltraInteract?

UltraInteract is a new large-scale, high-quality alignment dataset designed for complex reasoning tasks. It includes preference trees with diverse reasoning chains, multi-turn interaction trajectories, and pairwise data for preference learning.

How does Eurus compare to GPT-3.5 Turbo?

Eurus-70B beats GPT-3.5 Turbo in reasoning through comprehensive benchmarking across 12 tests covering five tasks, achieving higher accuracy on challenging benchmarks like LeetCode and TheoremQA.

Is Eurus open-source?

Yes, both the models and the UltraInteract dataset are available publicly, as mentioned in the paper.