Neura Market
SciBench logo

SciBench

Free

benchmark designed to evaluate large language models (LLMs) on solving complex, college-level scientific problems from domains like chemistry, physics, and mathematics.

FreeFree tier
Inputs: text, imageOutputs: text
Type
Open Source
Visit Website GitHub (38 stars)

About SciBench

SciBench is a comprehensive benchmark suite designed to evaluate the reasoning capabilities of Large Language Models (LLMs) on complex, college-level scientific problems. Curated from widely-used textbooks in chemistry, physics, and mathematics, the dataset consists of open-ended, free-response questions that require multi-step reasoning, domain-specific knowledge retrieval, and advanced numerical computation (e.g., calculus, differential equations). The problems also incorporate visual elements such as figures, requiring integration of textual and visual information. The benchmark includes detailed solutions for fine-grained error analysis. Presented at ICLR 2024 as an Oral paper, the study benchmarks open-source and proprietary LLMs under various prompting strategies, finding that even the best model achieves only 48.96% accuracy. The analysis categorizes errors into ten problem-solving abilities, revealing that no single prompting strategy universally outperforms others and that improvements in some skills may cause declines in others. SciBench aims to drive further advancements in LLM reasoning for scientific research and discovery.

Key Features

Curated dataset of collegiate-level problems from chemistry, physics, and mathematics textbooks
Open-ended, free-response questions requiring multi-step reasoning and complex computations
Includes visual elements (figures) that integrate textual and visual information
Detailed solutions provided for each problem to enable fine-grained error analysis
Benchmarks multiple open-source and proprietary LLMs under various prompting strategies
Categorizes LLM errors into ten distinct problem-solving abilities
Reveals overall best performance of only 48.96% accuracy across models
Open-source dataset and code with a public leaderboard
Zero-shot evaluation setting used for reported results

Pros & Cons

Pros
  • Covers three core scientific domains (chemistry, physics, mathematics) with college-level difficulty
  • Includes both textual and visual elements, reflecting real-world problem complexity
  • Provides detailed solutions for deep error analysis and model improvement
  • Open-source dataset and code, supporting reproducibility and community contributions
  • Accepted as ICLR 2024 Oral, indicating high academic quality and significance
  • Exposes current LLM limitations, setting a clear benchmark for progress
Cons
  • Limited to textbook-style problems, which may not fully represent real-world scientific research or experimental contexts
  • Only covers three domains; other scientific fields are not included
  • The benchmark may be computationally expensive due to the complexity of problems and multiple evaluations required
  • No standard API for automated evaluation; users must run their own models and submit results to leaderboard

Best For

Evaluating and comparing LLMs on complex scientific reasoning tasksAnalyzing specific limitations of LLMs in multi-step problem solving and domain knowledge retrievalBenchmarking new prompting strategies for scientific problem solvingTraining and fine-tuning LLMs to improve scientific reasoning capabilitiesAcademic research on AI reasoning and educational assessment

FAQ

What is SciBench?
SciBench is a benchmark suite for evaluating large language models (LLMs) on college-level scientific problems from chemistry, physics, and mathematics. It contains open-ended, free-response questions requiring multi-step reasoning and domain knowledge, and includes visual elements like figures.
What types of problems are in SciBench?
Problems are curated from widely-used college textbooks in chemistry, physics, and mathematics. They are open-ended and demand multi-step reasoning, understanding of scientific concepts, retrieval of equations/theorems, and complex numeric computation (e.g., calculus, differential equations). Many problems include visual elements (figures).
How were LLMs evaluated in the study?
The study benchmarked representative open-source and proprietary LLMs under various prompting strategies (e.g., zero-shot, few-shot). The main reported results are under zero-shot setting. Performance is measured as accuracy on the dataset, with detailed error analysis categorizing failures into ten problem-solving abilities.
What were the main findings?
Current LLMs achieve at most 48.96% overall accuracy. No single prompting strategy significantly outperforms others; some strategies that improve certain skills may cause declines in others. Errors were categorized into ten ability areas, highlighting specific weaknesses in scientific reasoning.