reasoning benchmark · included in ECI
BBH
BBH benchmark (score column: Average).
Results
45
Random baseline
25.0%
Score ceiling
100%
Released
17 Oct 2022
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | Gemini 1.5 Pro (May 2024) Google DeepMind | 89.2% | — | 14 May 2024 | — | Source ↗ | |
| 2 | DeepSeek V3 Open source DeepSeek | 87.5% | — | 26 Dec 2024 | — | Source ↗ | |
| 3 | Gemini 1.5 Pro (Feb 2024) Google DeepMind | 84.0% | — | 15 Feb 2024 | — | Source ↗ | |
| 4 | Llama 3.1-405B Open source Meta AI | 82.9% | — | 23 Jul 2024 | — | Source ↗ | |
| 5 | phi-3-medium 14B Open source Microsoft | 81.4% | — | 23 Apr 2024 | — | Source ↗ | |
| 6 | Qwen2.5-72B Open source Alibaba | 79.8% | — | 19 Sep 2024 | — | Source ↗ | |
| 7 | phi-3-small 7.4B Open source Microsoft | 79.1% | — | 23 Apr 2024 | — | Source ↗ | |
| 8 | DeepSeek-V2 (MoE-236B, May 2024) Open source DeepSeek | 78.8% | — | 7 May 2024 | — | Source ↗ | |
| 9 | GPT-4 (Jun 2023) OpenAI | 75.1% | — | 13 Jun 2023 | — | Source ↗ | |
| 10 | phi-3-mini 3.8B Open source Microsoft | 71.7% | — | 23 Apr 2024 | — | Source ↗ | |
| 11 | Yi-34B Open source 01.AI | 71.7% | — | 2 Nov 2023 | — | Source ↗ | |
| 12 | Stable Beluga 2 Open source Stability AI | 69.3% | — | 20 Jul 2023 | — | Source ↗ | |
| 13 | Llama 2-70B Open source Meta AI | 64.9% | — | 18 Jul 2023 | — | Source ↗ | |
| 14 | GPT-3.5 Turbo (June 2023) OpenAI | 61.6% | — | 13 Jun 2023 | — | Source ↗ | |
| 15 | Phi-2 Open source Microsoft | 59.4% | — | 12 Dec 2023 | — | Source ↗ | |
| 16 | Nemotron-4 15B NVIDIA | 58.7% | — | 26 Feb 2024 | — | Source ↗ | |
| 17 | LLaMA-65B Open source Meta AI | 58.4% | — | 24 Feb 2023 | — | Source ↗ | |
| 18 | Llama 2-13B Open source Meta AI | 58.2% | — | 18 Jul 2023 | — | Source ↗ | |
| 19 | Mistral 7B v0.1 Open source Mistral AI | 56.1% | — | 27 Sep 2023 | — | Source ↗ | |
| 20 | Gemma 7B Open source Google DeepMind | 55.1% | — | 21 Feb 2024 | — | Source ↗ | |
| 21 | Qwen-14B Open source Alibaba | 55.0% | — | 24 Sep 2023 | — | Source ↗ | |
| 22 | internlm-20b | 52.5% | — | 18 Sep 2023 | — | Source ↗ | |
| 23 | LLaMA-33B Open source Meta AI | 50.0% | — | 24 Feb 2023 | — | Source ↗ | |
| 24 | Baichuan2-13B Open source Baichuan | 49.0% | — | 6 Sep 2023 | — | Source ↗ | |
| 25 | Yi 6B Open source 01.AI | 47.2% | — | 22 Nov 2023 | — | Source ↗ | |
| 26 | Baichuan2-13B-Chat | 47.2% | — | 6 Sep 2023 | — | Source ↗ | |
| 27 | Qwen-7B Open source Alibaba | 45.0% | — | 28 Sep 2023 | — | Source ↗ | |
| 28 | Llama 2-34B Meta AI | 44.1% | — | 18 Jul 2023 | — | Source ↗ | |
| 29 | vicuna-13b-v1.1 | 43.0% | — | 12 Apr 2023 | — | Source ↗ | |
| 30 | Baichuan 1-13B Open source Baichuan | 43.0% | — | 11 Jul 2023 | — | Source ↗ | |
| 31 | internlm-chat-20b | 42.4% | — | 17 Sep 2023 | — | Source ↗ | |
| 32 | Baichuan 2-7B Open source Baichuan | 41.6% | — | 20 Sep 2023 | — | Source ↗ | |
| 33 | Llama 2-7B Open source Meta AI | 39.2% | — | 18 Jul 2023 | — | Source ↗ | |
| 34 | MPT-30B Open source MosaicML | 38.0% | — | 22 Jun 2023 | — | Source ↗ | |
| 35 | LLaMA-13B Open source Meta AI | 37.9% | — | 24 Feb 2023 | — | Source ↗ | |
| 36 | Falcon-40B Open source Technology Innovation Institute | 37.1% | — | 25 May 2023 | — | Source ↗ | |
| 37 | internlm-7b | 37.0% | — | 5 Jul 2023 | — | Source ↗ | |
| 38 | MPT-7B Open source MosaicML | 35.6% | — | 5 May 2023 | — | Source ↗ | |
| 39 | Gemma 2B Open source Google DeepMind | 35.2% | — | 21 Feb 2024 | — | Source ↗ | |
| 40 | INTELLECT-1 Prime Intellect,Hugging Face,Arcee AI | 34.8% | — | 29 Nov 2024 | — | Source ↗ | |
| 41 | chatglm2-6b | 33.7% | — | 24 Jun 2023 | — | Source ↗ | |
| 42 | LLaMA-7B Open source Meta AI | 33.5% | — | 24 Feb 2023 | — | Source ↗ | |
| 43 | Baichuan1-7B Open source Baichuan | 32.5% | — | 1 Jun 2023 | — | Source ↗ | |
| 44 | Falcon-7B Open source Technology Innovation Institute | 28.8% | — | 24 Apr 2023 | — | Source ↗ | |
| 45 | Qwen-1_8B | 28.2% | 8B | 30 Nov 2023 | — | Source ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.