HELM

HELM (Holistic Evaluation of Language Models) is an open-source Python framework developed by the Center for Research on Foundation Models (CRFM) at Stanford University. It enables holistic, reproducible, and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. The framework provides standardized datasets and benchmarks (e.g., MMLU-Pro, GPQA, IFEval, WildBench), a unified interface to access models from multiple providers (OpenAI, Anthropic, Google Gemini, and others), and metrics that measure not only accuracy but also efficiency, bias, and toxicity. HELM includes a web UI for inspecting individual prompts and responses, as well as a web leaderboard for comparing results across models and benchmarks. It supports official leaderboards such as HELM Capabilities, HELM Safety, and VHELM (Holistic Evaluation of Vision-Language Models). The package can be installed via pip and used through command-line tools for running, summarizing, and visualizing evaluations.