Neura Market
We-Math logo

We-Math

Free

a benchmark that evaluates large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning.

FreeFree tier
Inputs: image, textOutputs: text
Type
Open Source
Visit Website GitHub (38 stars)

About We-Math

We-Math is a benchmark designed to evaluate large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning. It goes beyond end-to-end performance by decomposing composite visual math problems into sub-problems based on hierarchical knowledge concepts. The benchmark includes 6.5K visual math problems spanning 67 knowledge concepts across 5 granularity levels, and introduces a four-dimensional metric (Insufficient Knowledge, Inadequate Generalization, Complete Mastery, Rote Memorization) to assess LMM reasoning processes. Evaluations reveal that GPT-4o transitions from knowledge insufficiency to generalization issues, while other LMMs rely more on rote memorization. The benchmark is openly available with code, dataset, and leaderboard.

Key Features

6.5K visual math problems spanning 67 hierarchical knowledge concepts and 5 layers of granularity
Decomposes composite problems into sub-problems based on knowledge concepts
Four-dimensional evaluation metric: IK, IG, CM, RM for hierarchical reasoning assessment
Knowledge-based reasoning evaluation beyond end-to-end performance
Knowledge concept augmentation using descriptions from Wikipedia and textbooks
Reveals negative correlation between solving step count and problem-specific performance
Identifies transition of GPT-4o from IK to IG, and rote memorization in other LMMs

Pros & Cons

Pros
  • Provides multi-dimensional evaluation of reasoning beyond simple accuracy
  • Hierarchical knowledge structure aligned with real textbook concepts
  • Knowledge decomposition enables detailed diagnosis of LMM issues
  • Open-source benchmark with available code, dataset, and leaderboard
  • First benchmark to explore problem-solving principles in LMMs
Cons
  • Primarily limited to visual mathematical reasoning from textbook concepts, may not generalize to other domains
  • Requires careful decomposition of problems which may be resource-intensive

Best For

Evaluating large multimodal models (LMMs) on visual mathematical reasoningAssessing LMMs' problem-solving principles and knowledge generalizationBenchmarking progress of LMMs towards human-like reasoningIdentifying specific weaknesses like insufficient knowledge or rote memorization in LMM reasoning

FAQ

What is We-Math?
We-Math is a benchmark for evaluating large multimodal models (LMMs) on human-like mathematical reasoning, focusing on problem-solving principles rather than just end-to-end accuracy.
How many problems does We-Math include?
It includes 6.5K visual math problems spanning 67 hierarchical knowledge concepts across 5 layers of granularity.
What metrics does We-Math use?
The benchmark introduces a four-dimensional metric: Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM).
Who created We-Math?
It was created by researchers from Beijing University of Posts and Telecommunications, Wechat/Tencent Inc., Huazhong University of Science and Technology, and Beijing Institute of Technology.
Is We-Math open source?
Yes, the code, dataset, and leaderboard are publicly available on GitHub and Hugging Face.