knowledge benchmark · included in ECI
MMLU
MMLU benchmark (score column: EM).
Results
119
Random baseline
25.0%
Score ceiling
100%
Released
7 Sep 2020
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | GPT-4o (Nov 2024) OpenAI | 88.1% | — | 20 Nov 2024 | — | Source ↗ | |
| 2 | Claude 3.5 Sonnet (Oct 2024) Anthropic | 87.3% | — | 22 Oct 2024 | — | Source ↗ | |
| 3 | DeepSeek V3 Open source DeepSeek | 87.2% | — | 26 Dec 2024 | — | Source ↗ | |
| 4 | Gemini 1.5 Pro (Sept 2024) Google DeepMind | 86.9% | — | 24 Sep 2024 | — | Source ↗ | |
| 5 | Claude 3.5 Sonnet (Jun 2024) Anthropic | 86.5% | — | 20 Jun 2024 | — | Source ↗ | |
| 6 | GPT-4 (Mar 2023) OpenAI | 86.4% | — | 14 Mar 2023 | — | Source ↗ | |
| 7 | Llama 3.3 70B Open source Meta AI | 86.3% | — | 6 Dec 2024 | — | Source ↗ | |
| 8 | Gemini 1.5 Pro (May 2024) Google DeepMind | 85.9% | — | 14 May 2024 | — | Source ↗ | |
| 9 | Qwen2.5-72B Open source Alibaba | 85.3% | — | 19 Sep 2024 | — | Source ↗ | |
| 10 | Phi 4 Open source Microsoft | 84.8% | — | 12 Dec 2024 | — | Source ↗ | |
| 11 | Claude 3 Opus Anthropic | 84.6% | — | 29 Feb 2024 | — | Source ↗ | |
| 12 | Llama 3.1-405B Open source Meta AI | 84.5% | — | 23 Jul 2024 | — | Source ↗ | |
| 13 | GPT-4o (Aug 2024) OpenAI | 84.3% | — | 6 Aug 2024 | — | Source ↗ | |
| 14 | GPT-4o (May 2024) OpenAI | 84.2% | — | 13 May 2024 | — | Source ↗ | |
| 15 | Gemini 1.5 Pro (Feb 2024) Google DeepMind | 82.7% | — | 15 Feb 2024 | — | Source ↗ | |
| 16 | Qwen2-72B Open source Alibaba | 82.4% | — | 7 Jun 2024 | — | Source ↗ | |
| 17 | GPT-4 (Jun 2023) OpenAI | 82.4% | — | 13 Jun 2023 | — | Source ↗ | |
| 18 | Amazon Nova Pro Amazon | 82.0% | — | 3 Dec 2024 | — | Source ↗ | |
| 19 | GPT-4o-mini OpenAI | 81.8% | — | 18 Jul 2024 | — | Source ↗ | |
| 20 | GPT-4 Turbo (Apr 2024) OpenAI | 81.3% | — | 9 Apr 2024 | — | Source ↗ | |
| 21 | Llama 3.2 90B Open source Meta AI | 80.3% | — | 24 Sep 2024 | — | Source ↗ | |
| 22 | Llama 3.1-70B Open source Meta AI | 80.1% | — | 23 Jul 2024 | — | Source ↗ | |
| 23 | Mistral Large 2 (Jul 2024) Open source Mistral AI | 80.0% | — | 24 Jul 2024 | — | Source ↗ | |
| 24 | Qwen2.5-14B Open source Alibaba | 79.9% | — | 19 Sep 2024 | — | Source ↗ | |
| 25 | Gemini 2.0 Flash (Dec 2024) Google DeepMind,Google | 79.7% | — | 11 Dec 2024 | — | Source ↗ | |
| 26 | GPT-4 Turbo (Nov 2023) OpenAI | 79.6% | — | 25 Jan 2024 | — | Source ↗ | |
| 27 | Yi-Large 01.AI | 79.3% | — | 13 May 2024 | — | Source ↗ | |
| 28 | Llama 3-70B Open source Meta AI | 79.3% | — | 18 Apr 2024 | — | Source ↗ | |
| 29 | Qwen2.5-Coder-32B Open source Alibaba | 79.1% | — | 18 Sep 2024 | — | Source ↗ | |
| 30 | Claude 2 Anthropic | 78.5% | — | 11 Jul 2023 | — | Source ↗ | |
| 31 | DeepSeek-V2 (MoE-236B, May 2024) Open source DeepSeek | 78.4% | — | 7 May 2024 | — | Source ↗ | |
| 32 | phi-3-medium 14B Open source Microsoft | 78.0% | — | 23 Apr 2024 | — | Source ↗ | |
| 33 | Gemini 1.5 Flash (May 2024) Google DeepMind | 77.9% | — | 23 May 2024 | — | Source ↗ | |
| 34 | Mixtral 8x22B Open source Mistral AI | 77.8% | — | 17 Apr 2024 | — | Source ↗ | |
| 35 | Amazon Nova Lite Amazon | 77.0% | — | 3 Dec 2024 | — | Source ↗ | |
| 36 | Claude 1.3 Anthropic | 77.0% | — | 18 Apr 2023 | — | Source ↗ | |
| 37 | Yi-34B Open source 01.AI | 76.3% | — | 2 Nov 2023 | — | Source ↗ | |
| 38 | Claude 3 Sonnet Anthropic | 75.9% | — | 29 Feb 2024 | — | Source ↗ | |
| 39 | Gemma 2 27B Open source Google | 75.7% | — | 24 Jun 2024 | — | Source ↗ | |
| 40 | phi-3-small 7.4B Open source Microsoft | 75.7% | — | 23 Apr 2024 | — | Source ↗ | |
| 41 | Qwen2.5-Coder-14B | 75.2% | — | 18 Sep 2024 | — | Source ↗ | |
| 42 | Qwen1.5-32B Open source Alibaba | 74.4% | — | 3 Apr 2024 | — | Source ↗ | |
| 43 | Claude 3.5 Haiku (Oct 2024) Anthropic | 74.3% | — | 22 Oct 2024 | — | Source ↗ | |
| 44 | Gemini 1.5 Flash (Sep 2024) Google DeepMind | 73.9% | — | 24 Sep 2024 | — | Source ↗ | |
| 45 | Claude 3 Haiku Anthropic | 73.8% | — | 7 Mar 2024 | — | Source ↗ | |
| 46 | Claude 2.1 Anthropic | 73.5% | — | 21 Nov 2023 | — | Source ↗ | |
| 47 | Claude Instant Anthropic | 73.4% | — | 9 Aug 2023 | — | Source ↗ | |
| 48 | Qwen2.5-7B Open source Alibaba | 72.9% | — | 19 Sep 2024 | — | Source ↗ | |
| 49 | Inflection-1 Inflection AI | 72.7% | — | 22 Jun 2023 | — | Source ↗ | |
| 50 | Gemma 2 9B Open source Google DeepMind | 72.1% | — | 24 Jun 2024 | — | Source ↗ | |
| 51 | GPT-3.5 Turbo (Nov 2023) OpenAI | 71.4% | — | 6 Nov 2023 | — | Source ↗ | |
| 52 | Amazon Nova Micro Amazon | 70.8% | — | 3 Dec 2024 | — | Source ↗ | |
| 53 | Mixtral 8x7B Open source Mistral AI | 70.6% | — | 11 Dec 2023 | — | Source ↗ | |
| 54 | Falcon-180B Open source Technology Innovation Institute | 70.6% | — | 6 Sep 2023 | — | Source ↗ | |
| 55 | Gemini 1.0 Pro Google DeepMind | 70.0% | — | 13 Dec 2023 | — | Source ↗ | |
| 56 | Llama 2-70B Open source Meta AI | 69.9% | — | 18 Jul 2023 | — | Source ↗ | |
| 57 | Command R+ Open source Cohere,Cohere Labs (formerly Cohere for AI) | 69.4% | — | 30 Aug 2024 | — | Source ↗ | |
| 58 | GPT-3.5 Turbo (June 2023) OpenAI | 68.9% | — | 13 Jun 2023 | — | Source ↗ | |
| 59 | phi-3-mini 3.8B Open source Microsoft | 68.8% | — | 23 Apr 2024 | — | Source ↗ | |
| 60 | Llama 3-8B Open source Meta AI | 68.8% | — | 18 Apr 2024 | — | Source ↗ | |
| 61 | Mistral Large mistralai | 68.8% | — | 26 Feb 2024 | — | Source ↗ | |
| 62 | mistral-small-2402 | 68.7% | — | 26 Feb 2024 | — | Source ↗ | |
| 63 | Qwen1.5-14B Open source Alibaba | 68.6% | — | 4 Feb 2024 | — | Source ↗ | |
| 64 | Stable Beluga 2 Open source Stability AI | 68.6% | — | 20 Jul 2023 | — | Source ↗ | |
| 65 | Yi-9B | 68.4% | — | 1 Mar 2024 | — | Source ↗ | |
| 66 | Qwen2.5-Coder (7B) Open source Alibaba | 68.0% | — | 18 Sep 2024 | — | Source ↗ | |
| 67 | GPT-3.5 Turbo (Jan 2024) OpenAI | 67.3% | — | 25 Jan 2024 | — | Source ↗ | |
| 68 | Qwen-14B Open source Alibaba | 66.3% | — | 24 Sep 2023 | — | Source ↗ | |
| 69 | Gemma 7B Open source Google DeepMind | 66.1% | — | 21 Feb 2024 | — | Source ↗ | |
| 70 | c4ai-command-r-08-2024 | 65.2% | — | 30 Aug 2024 | — | Source ↗ | |
| 71 | StarCoder 2 15B Open source Hugging Face,ServiceNow,NVIDIA,BigCode | 64.1% | — | 20 Feb 2024 | — | Source ↗ | |
| 72 | Yi 6B Open source 01.AI | 64.0% | — | 22 Nov 2023 | — | Source ↗ | |
| 73 | LLaMA-65B Open source Meta AI | 63.4% | — | 24 Feb 2023 | — | Source ↗ | |
| 74 | Qwen1.5-7B Open source Alibaba | 62.6% | — | 4 Feb 2024 | — | Source ↗ | |
| 75 | Llama 2-34B Meta AI | 62.6% | — | 18 Jul 2023 | — | Source ↗ | |
| 76 | Mistral 7B v0.2 Open source Mistral AI | 62.5% | — | 11 Dec 2023 | — | Source ↗ | |
| 77 | Mistral 7B v0.1 Open source Mistral AI | 62.5% | — | 27 Sep 2023 | — | Source ↗ | |
| 78 | DeepSeek-Coder-V2-Lite-Base | 60.5% | — | 13 Jun 2024 | — | Source ↗ | |
| 79 | Mistral 7B v0.3 Open source Mistral AI | 59.9% | — | 27 May 2024 | — | Source ↗ | |
| 80 | Baichuan2-13B Open source Baichuan | 59.2% | — | 6 Sep 2023 | — | Source ↗ | |
| 81 | Nemotron-4 15B NVIDIA | 58.7% | — | 26 Feb 2024 | — | Source ↗ | |
| 82 | LLaMA-33B Open source Meta AI | 58.7% | — | 24 Feb 2023 | — | Source ↗ | |
| 83 | Falcon 2 11B Open source Technology Innovation Institute | 58.4% | — | 9 May 2024 | — | Source ↗ | |
| 84 | Phi-2 Open source Microsoft | 58.4% | — | 12 Dec 2023 | — | Source ↗ | |
| 85 | internlm-chat-20b | 57.4% | — | 17 Sep 2023 | — | Source ↗ | |
| 86 | Falcon-40B Open source Technology Innovation Institute | 56.9% | — | 25 May 2023 | — | Source ↗ | |
| 87 | Llama 3.2 11B Open source Meta AI | 56.5% | — | 24 Sep 2024 | — | Source ↗ | |
| 88 | Llama 3.1-8B Open source Meta AI | 56.1% | — | 23 Jul 2024 | — | Source ↗ | |
| 89 | Llama 2-13B Open source Meta AI | 55.6% | — | 18 Jul 2023 | — | Source ↗ | |
| 90 | Baichuan2-13B-Chat | 55.1% | — | 6 Sep 2023 | — | Source ↗ | |
| 91 | Baichuan 2-7B Open source Baichuan | 54.2% | — | 20 Sep 2023 | — | Source ↗ | |
| 92 | Qwen2.5-Coder (1.5B) Open source Alibaba | 53.6% | — | 18 Sep 2024 | — | Source ↗ | |
| 93 | Baichuan 1-13B Open source Baichuan | 51.6% | — | 11 Jul 2023 | — | Source ↗ | |
| 94 | internlm-7b | 51.0% | — | 5 Jul 2023 | — | Source ↗ | |
| 95 | INTELLECT-1 Prime Intellect,Hugging Face,Arcee AI | 49.9% | — | 29 Nov 2024 | — | Source ↗ | |
| 96 | chatglm2-6b | 47.9% | — | 24 Jun 2023 | — | Source ↗ | |
| 97 | MPT-30B Open source MosaicML | 47.9% | — | 22 Jun 2023 | — | Source ↗ | |
| 98 | LLaMA-13B Open source Meta AI | 47.7% | — | 24 Feb 2023 | — | Source ↗ | |
| 99 | Llama 2-7B Open source Meta AI | 45.8% | — | 18 Jul 2023 | — | Source ↗ | |
| 100 | Qwen-7B Open source Alibaba | 45.0% | — | 28 Sep 2023 | — | Source ↗ | |
| 101 | Gemma 2B Open source Google DeepMind | 42.3% | — | 21 Feb 2024 | — | Source ↗ | |
| 102 | Baichuan1-7B Open source Baichuan | 42.3% | — | 1 Jun 2023 | — | Source ↗ | |
| 103 | Qwen2.5-Coder-0.5B | 42.0% | — | 18 Sep 2024 | — | Source ↗ | |
| 104 | CodeQwen1.5-7B | 40.5% | — | 15 Apr 2024 | — | Source ↗ | |
| 105 | DeepSeek Coder 33B Open source DeepSeek,Peking University | 39.4% | — | 2 Nov 2023 | — | Source ↗ | |
| 106 | StarCoder 2 7B Open source Hugging Face,ServiceNow,NVIDIA,BigCode | 38.8% | — | 20 Feb 2024 | — | Source ↗ | |
| 107 | Phi-1.5 Open source Microsoft | 37.6% | 5 | 11 Sep 2023 | — | Source ↗ | |
| 108 | StarCoder 2 3B Open source Hugging Face,ServiceNow,NVIDIA,BigCode | 36.6% | — | 22 Feb 2024 | — | Source ↗ | |
| 109 | DeepSeek Coder 6.7B Open source DeepSeek,Peking University | 36.4% | — | 2 Nov 2023 | — | Source ↗ | |
| 110 | XGen-7B Open source Salesforce | 36.3% | — | 27 Jun 2023 | — | Source ↗ | |
| 111 | LLaMA-7B Open source Meta AI | 35.6% | — | 24 Feb 2023 | — | Source ↗ | |
| 112 | Falcon-7B Open source Technology Innovation Institute | 35.0% | — | 24 Apr 2023 | — | Source ↗ | |
| 113 | MPT-7B Open source MosaicML | 30.8% | — | 5 May 2023 | — | Source ↗ | |
| 114 | open_llama_7b | 29.9% | 7b | 7 Jun 2023 | — | Source ↗ | |
| 115 | Qwen-1_8B | 28.2% | 8B | 30 Nov 2023 | — | Source ↗ | |
| 116 | RedPajama-INCITE-7B-Base | 26.3% | — | 4 May 2023 | — | Source ↗ | |
| 117 | Dolly 2.0-12b Open source Databricks | 26.2% | — | 11 Apr 2023 | — | Source ↗ | |
| 118 | Cerebras-GPT-13B Open source Cerebras Systems | 26.2% | — | 20 Mar 2023 | — | Source ↗ | |
| 119 | DeepSeek Coder 1.3B Open source DeepSeek,Peking University | 25.8% | — | 2 Nov 2023 | — | Source ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.