other benchmark · included in ECI
ARC AI2
ARC AI2 benchmark (score column: Challenge score).
Results
67
Random baseline
25.0%
Score ceiling
100%
Released
14 Mar 2018
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | DeepSeek V3 Open source DeepSeek | 95.3% | — | 26 Dec 2024 | — | Source ↗ | |
| 2 | Llama 3.1-405B Open source Meta AI | 95.3% | — | 23 Jul 2024 | — | Source ↗ | |
| 3 | Qwen2.5-72B Open source Alibaba | 94.5% | — | 19 Sep 2024 | — | Source ↗ | |
| 4 | DeepSeek-V2 (MoE-236B, May 2024) Open source DeepSeek | 92.2% | — | 7 May 2024 | — | Source ↗ | |
| 5 | phi-3-medium 14B Open source Microsoft | 91.6% | — | 23 Apr 2024 | — | Source ↗ | |
| 6 | phi-3-small 7.4B Open source Microsoft | 90.7% | — | 23 Apr 2024 | — | Source ↗ | |
| 7 | GPT-3.5 Turbo (Nov 2023) OpenAI | 87.4% | — | 6 Nov 2023 | — | Source ↗ | |
| 8 | Mixtral 8x7B Open source Mistral AI | 87.3% | — | 11 Dec 2023 | — | Source ↗ | |
| 9 | Claude Instant Anthropic | 86.3% | — | 9 Aug 2023 | — | Source ↗ | |
| 10 | Stable Beluga 2 Open source Stability AI | 86.1% | — | 20 Jul 2023 | — | Source ↗ | |
| 11 | phi-3-mini 3.8B Open source Microsoft | 84.9% | — | 23 Apr 2024 | — | Source ↗ | |
| 12 | Qwen-14B Open source Alibaba | 84.4% | — | 24 Sep 2023 | — | Source ↗ | |
| 13 | Llama 3-8B Open source Meta AI | 82.8% | — | 18 Apr 2024 | — | Source ↗ | |
| 14 | internlm-20b | 81.7% | — | 18 Sep 2023 | — | Source ↗ | |
| 15 | Mistral 7B v0.1 Open source Mistral AI | 78.6% | — | 27 Sep 2023 | — | Source ↗ | |
| 16 | Gemma 7B Open source Google DeepMind | 78.3% | — | 21 Feb 2024 | — | Source ↗ | |
| 17 | Llama 2-70B Open source Meta AI | 78.3% | — | 18 Jul 2023 | — | Source ↗ | |
| 18 | Phi-2 Open source Microsoft | 75.9% | — | 12 Dec 2023 | — | Source ↗ | |
| 19 | Qwen-7B Open source Alibaba | 75.3% | — | 28 Sep 2023 | — | Source ↗ | |
| 20 | Qwen2.5-Coder-32B Open source Alibaba | 70.5% | — | 18 Sep 2024 | — | Source ↗ | |
| 21 | internlm-7b | 69.5% | — | 5 Jul 2023 | — | Source ↗ | |
| 22 | LLaMA-65B Open source Meta AI | 69.5% | — | 24 Feb 2023 | — | Source ↗ | |
| 23 | PaLM 2-L | 69.2% | — | 17 May 2023 | — | Source ↗ | |
| 24 | Falcon-180B Open source Technology Innovation Institute | 67.8% | — | 6 Sep 2023 | — | Source ↗ | |
| 25 | LLaMA-33B Open source Meta AI | 67.5% | — | 24 Feb 2023 | — | Source ↗ | |
| 26 | Qwen2.5-Coder-14B | 66.0% | — | 18 Sep 2024 | — | Source ↗ | |
| 27 | PaLM 2-M | 64.9% | — | 17 May 2023 | — | Source ↗ | |
| 28 | DeepSeek-Coder-V2 236B Open source DeepSeek | 64.3% | — | 17 Jun 2024 | — | Source ↗ | |
| 29 | Falcon-40B Open source Technology Innovation Institute | 61.9% | — | 25 May 2023 | — | Source ↗ | |
| 30 | chatglm2-6b | 61.0% | — | 24 Jun 2023 | — | Source ↗ | |
| 31 | Qwen2.5-Coder (7B) Open source Alibaba | 60.9% | — | 18 Sep 2024 | — | Source ↗ | |
| 32 | Llama 2-13B Open source Meta AI | 60.3% | — | 18 Jul 2023 | — | Source ↗ | |
| 33 | PaLM 2-S | 59.6% | — | 17 May 2023 | — | Source ↗ | |
| 34 | DeepSeek-Coder-V2-Lite-Base | 57.3% | — | 13 Jun 2024 | — | Source ↗ | |
| 35 | Yi-9B | 55.6% | — | 1 Mar 2024 | — | Source ↗ | |
| 36 | Nemotron-4 15B NVIDIA | 55.5% | — | 26 Feb 2024 | — | Source ↗ | |
| 37 | INTELLECT-1 Prime Intellect,Hugging Face,Arcee AI | 54.5% | — | 29 Nov 2024 | — | Source ↗ | |
| 38 | Llama 2-34B Meta AI | 54.5% | — | 18 Jul 2023 | — | Source ↗ | |
| 39 | Qwen-1_8B | 53.2% | 8B | 30 Nov 2023 | — | Source ↗ | |
| 40 | Qwen2.5-Coder-3B | 52.9% | — | 18 Sep 2024 | — | Source ↗ | |
| 41 | LLaMA-13B Open source Meta AI | 52.7% | — | 24 Feb 2023 | — | Source ↗ | |
| 42 | MPT-30B Open source MosaicML | 50.6% | — | 22 Jun 2023 | — | Source ↗ | |
| 43 | Yi 6B Open source 01.AI | 50.3% | — | 22 Nov 2023 | — | Source ↗ | |
| 44 | Falcon-7B Open source Technology Innovation Institute | 47.9% | — | 24 Apr 2023 | — | Source ↗ | |
| 45 | LLaMA-7B Open source Meta AI | 47.6% | — | 24 Feb 2023 | — | Source ↗ | |
| 46 | StarCoder 2 15B Open source Hugging Face,ServiceNow,NVIDIA,BigCode | 47.2% | — | 20 Feb 2024 | — | Source ↗ | |
| 47 | Llama 2-7B Open source Meta AI | 45.9% | — | 18 Jul 2023 | — | Source ↗ | |
| 48 | Qwen2.5-Coder (1.5B) Open source Alibaba | 45.2% | — | 18 Sep 2024 | — | Source ↗ | |
| 49 | Phi-1.5 Open source Microsoft | 44.4% | 5 | 11 Sep 2023 | — | Source ↗ | |
| 50 | vicuna-13b-v1.1 | 43.2% | — | 12 Apr 2023 | — | Source ↗ | |
| 51 | MPT-7B Open source MosaicML | 42.6% | — | 5 May 2023 | — | Source ↗ | |
| 52 | DeepSeek Coder 33B Open source DeepSeek,Peking University | 42.2% | — | 2 Nov 2023 | — | Source ↗ | |
| 53 | Gemma 2B Open source Google DeepMind | 42.1% | — | 21 Feb 2024 | — | Source ↗ | |
| 54 | XGen-7B Open source Salesforce | 41.2% | — | 27 Jun 2023 | — | Source ↗ | |
| 55 | Dolly 2.0-12b Open source Databricks | 39.6% | — | 11 Apr 2023 | — | Source ↗ | |
| 56 | RedPajama-INCITE-7B-Base | 39.1% | — | 4 May 2023 | — | Source ↗ | |
| 57 | StarCoder 2 7B Open source Hugging Face,ServiceNow,NVIDIA,BigCode | 38.7% | — | 20 Feb 2024 | — | Source ↗ | |
| 58 | open_llama_7b | 38.7% | 7b | 7 Jun 2023 | — | Source ↗ | |
| 59 | Baichuan2-13B Open source Baichuan | 38.0% | — | 6 Sep 2023 | — | Source ↗ | |
| 60 | DeepSeek Coder 6.7B Open source DeepSeek,Peking University | 36.4% | — | 2 Nov 2023 | — | Source ↗ | |
| 61 | CodeQwen1.5-7B | 35.7% | — | 15 Apr 2024 | — | Source ↗ | |
| 62 | Qwen2.5-Coder-0.5B | 34.4% | — | 18 Sep 2024 | — | Source ↗ | |
| 63 | StarCoder 2 3B Open source Hugging Face,ServiceNow,NVIDIA,BigCode | 34.2% | — | 22 Feb 2024 | — | Source ↗ | |
| 64 | Baichuan 2-7B Open source Baichuan | 32.5% | — | 20 Sep 2023 | — | Source ↗ | |
| 65 | Cerebras-GPT-13B Open source Cerebras Systems | 32.4% | — | 20 Mar 2023 | — | Source ↗ | |
| 66 | stablelm-tuned-alpha-7b | 27.0% | — | 19 Apr 2023 | — | Source ↗ | |
| 67 | DeepSeek Coder 1.3B Open source DeepSeek,Peking University | 25.4% | — | 2 Nov 2023 | — | Source ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.