Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

Free

Scaling inference compute through repeated sampling of LLMs

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Large Language Monkeys: Scaling Inference Compute with Repeated Sampling

Large Language Monkeys is a research paper that explores scaling inference compute by repeatedly sampling candidate solutions from large language models. The study demonstrates that coverage—the fraction of problems solved by any generated sample—scales with the number of samples over four orders of magnitude, often following a log-linear relationship modeled by an exponentiated power law, suggesting inference-time scaling laws. In domains with automatic verification, such as coding and formal proofs, repeated sampling directly translates into improved performance. For example, on SWE-bench Lite, using DeepSeek-Coder-V2-Instruct increases issue resolution from 15.9% with one sample to 56% with 250 samples, outperforming the single-sample state-of-the-art of 43%. In domains without automatic verifiers, common selection methods like majority voting and reward models plateau beyond several hundred samples, failing to fully scale with the sample budget.

Key Features

Repeated sampling from a model to generate multiple candidate solutions

Coverage scaling over four orders of magnitude with log-linear relationship

Identifies inference-time scaling laws (exponentiated power law)

Effective for automatically verifiable domains (coding, formal proofs)

Demonstrated 56% issue resolution on SWE-bench Lite with 250 samples

Analysis of plateau behavior for majority voting and reward models in non-verifiable domains

Pros & Cons

Pros

Simple technique requiring no additional training or fine-tuning
Provides substantial performance improvements in verifiable domains
Reveals universal scaling laws for inference compute
Applicable to any pre-existing language model
Directly increases task coverage with more samples

Cons

Performance gain plateaus in domains without automatic verifiers
Requires sufficient compute budget for generating many samples
Selection methods like majority voting and reward models fail to fully scale with sample budget
Not suitable for tasks where ground truth verification is unavailable

Best For

Improving code generation accuracy via multiple sampled solutionsAutomated formal proof generation and verificationEnhancing performance on software engineering benchmarks (e.g., SWE-bench Lite)Studying scaling behavior of LLMs at inference timeComparing selection methods for sample collections (majority voting, reward models)

FAQ

What is the main idea behind Large Language Monkeys?

The paper explores scaling inference compute by repeatedly sampling candidate solutions from a language model, rather than limiting the model to a single attempt.

How does repeated sampling improve performance in coding tasks?

On SWE-bench Lite, the fraction of issues solved with DeepSeek-Coder-V2-Instruct increased from 15.9% with one sample to 56% with 250 samples, outperforming the single-sample state-of-the-art of 43%.

Does this technique work in domains without automatic verification?

In domains without automatic verifiers, common methods like majority voting and reward models plateau beyond several hundred samples and fail to fully scale with the sample budget.