InfiBench
Freea benchmark designed to evaluate large language models (LLMs) specifically in their ability to answer real-world coding-related questions.
FreeFree tier
Inputs: textOutputs: text
About InfiBench
InfiBench is a comprehensive benchmark for code large language models that evaluates model ability on answering freeform real-world questions in the code domain. It comprises 234 carefully selected high-quality Stack Overflow questions across 15 programming languages, using four types of model-free automatic metrics (keywords matching, blank filling, unit testing, and dialogue similarity). The benchmark is fully open source and continuously expanding, and has been used to systematically evaluate over 100 latest code LLMs, leading to novel insights.
Key Features
234 carefully selected high-quality Stack Overflow questions spanning 15 programming languages
Four types of model-free automatic metrics: keywords matching, blank filling, unit testing, and dialogue similarity
Systematic evaluation of over 100 latest code LLMs
Fully open source and continuously expanding
Uses best@10 evaluation metric with sampling temperature 0.2 and top p 0.9
Provides system prompt engineering for different question types
Pros & Cons
Pros
- Covers 15 programming languages with natural distribution from Stack Overflow
- Uses multiple automatic metrics that do not require human evaluation
- Based on real-world questions rather than synthetic tasks
- Open source and designed to be continuously expanded
- Systematic evaluation provides insights beyond code generation benchmarks
Cons
- Limited to 234 questions (though planned to expand)
- Only uses Stack Overflow questions, which may not cover all coding domains
- Automatic metrics may not fully capture semantic correctness
- Requires significant compute for best@10 sampling across models
Best For
Evaluating code large language models on real-world programming question answeringComparing model performance across multiple automatic metricsResearch on code understanding and question-answering capabilitiesBenchmarking for new code LLM releases and open-source model comparisonsAnalyzing LLM strengths and weaknesses in diverse programming languages
FAQ
What is InfiBench?
InfiBench is a comprehensive benchmark for code large language models that evaluates their ability to answer freeform real-world coding questions. It includes 234 high-quality Stack Overflow questions across 15 programming languages and uses four types of automatic metrics to measure response correctness.
What metrics does InfiBench use?
InfiBench integrates four types of model-free automatic metrics: keywords matching, blank filling, unit testing, and dialogue similarity.