# LLM benchmarks

Discover standard benchmarks used to evaluate large language models (LLMs). Includes HELM, MMLU, TruthfulQA, and more.

LLM benchmarks are standardized tests and datasets used to evaluate the performance of large language models. These benchmarks provide a way to compare models across tasks like reasoning, question answering, coding, and ethics.

## What are LLM benchmarks? {#what-are-llm-benchmarks}

Benchmarks are curated tasks designed to measure LLM capabilities in a consistent way. They often include:

- Multiple-choice questions
- Free-form answers to complex prompts
- Code generation tasks
- Ethical or factuality stress tests

These benchmarks are used in academic research, competitive leaderboards, and internal product evaluations.

## Why it matters in AI/ML {#why-it-matters-in-ai-ml}

Benchmarks help:

- Compare models from different providers (e.g., OpenAI, Anthropic, Cohere)
- Track progress in model capabilities
- Identify gaps in reasoning, safety, or factuality

They offer a shared language for researchers, builders, and buyers to evaluate model strengths and weaknesses.

## Common LLM benchmarks {#common-llm-benchmarks}

- **MMLU (Massive multitask language understanding):** Tests reasoning across 57 academic subjects
- **TruthfulQA:** Evaluates factual correctness and robustness to misinformation
- **HellaSwag:** Measures common-sense reasoning and plausibility
- **Big-Bench (BIG-bench):** Collaborative benchmark with 200+ tasks
- **GSM8K:** Math word problems for grade-school level reasoning
- **ARC (AI2 reasoning challenge):** Science questions from standardized tests
- **CodeEval / HumanEval:** For evaluating code generation tasks
- **HELM (Holistic evaluation of language models):** Framework for comparing models across multiple dimensions like fairness, robustness, and toxicity

## Limitations {#limitations}

- Benchmarks may not reflect real-world use cases
- Many models overfit to public benchmarks
- Results vary based on prompt formatting and inference parameters

## Related {#related}

- [LLM evaluation metrics](https://openlayer.com/glossary/llm-evaluation-metrics)
- [GenAI testing](https://www.openlayer.com/products/genai-testing)

*Use benchmarks alongside task-specific evaluation to get a full picture of LLM performance.*