coding benchmark · included in ECI
Aider polyglot
Code editing exercises in C++, Go, Java, JavaScript, Python and Rust.
Results
60
Random baseline
0.0%
Score ceiling
100%
Released
21 Dec 2024
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | GPT-5 OpenAI | 88.0% | high | 7 Aug 2025 | — | Source ↗ | |
| 2 | GPT-5 (medium) OpenAI | 86.7% | medium | 7 Aug 2025 | — | Source ↗ | |
| 3 | o3 Pro OpenAI | 84.9% | high | 10 Jun 2025 | — | Source ↗ | |
| 4 | Gemini 2.5 Pro Preview (Jun 2025) Google DeepMind | 83.1% | 32K | 5 Jun 2025 | — | Source ↗ | |
| 5 | GPT-5 (low) OpenAI | 81.3% | low | 7 Aug 2025 | — | Source ↗ | |
| 6 | o3 OpenAI | 81.3% | high | 16 Apr 2025 | — | Source ↗ | |
| 7 | Grok 4 xAI | 79.6% | — | 9 Jul 2025 | — | Source ↗ | |
| 8 | o3 (medium) OpenAI | 76.9% | medium | 16 Apr 2025 | — | Source ↗ | |
| 9 | o3 (unknown thinking) OpenAI | 76.9% | unknown | 16 Apr 2025 | — | Source ↗ | |
| 10 | DeepSeek V3.2 Open source DeepSeek | 74.2% | — | 1 Dec 2025 | — | Source ↗ | |
| 11 | DeepSeek V3.2 Exp Open source DeepSeek | 74.2% | thinking | 29 Sep 2025 | — | Source ↗ | |
| 12 | Gemini 2.5 Pro Preview (Mar 2025) Google DeepMind | 72.9% | — | 31 Mar 2025 | — | Source ↗ | |
| 13 | Gemini 2.5 Pro Exp (Mar 2025) Google DeepMind | 72.9% | — | 25 Mar 2025 | — | Source ↗ | |
| 14 | Claude Opus 4 Anthropic | 72.0% | 32K | 22 May 2025 | — | Source ↗ | |
| 15 | o4 Mini OpenAI | 72.0% | high | 16 Apr 2025 | — | Source ↗ | |
| 16 | DeepSeek-R1 (May 2025) Open source DeepSeek | 71.4% | — | 28 May 2025 | — | Source ↗ | |
| 17 | Claude 3.7 Sonnet (32k thinking) Anthropic | 64.9% | 32K | 24 Feb 2025 | — | Source ↗ | |
| 18 | o1 OpenAI | 61.7% | high | 17 Dec 2024 | — | Source ↗ | |
| 19 | Claude Sonnet 4 Anthropic | 61.3% | 32K | 22 May 2025 | — | Source ↗ | |
| 20 | Claude 3.7 Sonnet Anthropic | 60.4% | — | 24 Feb 2025 | — | Source ↗ | |
| 21 | o3 Mini OpenAI | 60.4% | high | 31 Jan 2025 | — | Source ↗ | |
| 22 | Qwen3-235B-A22B-Instruct (Jul 2025) Open source Alibaba | 59.6% | — | 25 Jul 2025 | — | Source ↗ | |
| 23 | Qwen3 235B A22B Open source Qwen | 59.6% | — | 28 Apr 2025 | — | Source ↗ | |
| 24 | Kimi K2 0905 (Novita) Open source Moonshot | 59.1% | — | 5 Sep 2025 | — | Source ↗ | |
| 25 | Kimi K2 Instruct Open source Moonshot | 59.1% | — | 12 Jul 2025 | — | Source ↗ | |
| 26 | R1 Open source DeepSeek | 56.9% | — | 20 Jan 2025 | — | Source ↗ | |
| 27 | Gemini 2.5 Flash (May 2025) Google DeepMind | 55.1% | 23K | 20 May 2025 | — | Source ↗ | |
| 28 | DeepSeek-V3 (Mar 2025) Open source DeepSeek | 55.1% | — | 24 Mar 2025 | — | Source ↗ | |
| 29 | o3-mini (medium) OpenAI | 53.8% | medium | 31 Jan 2025 | — | Source ↗ | |
| 30 | Grok 3 xAI | 53.3% | — | 9 Apr 2025 | — | Source ↗ | |
| 31 | GPT-4.1 OpenAI | 52.4% | — | 14 Apr 2025 | — | Source ↗ | |
| 32 | Claude 3.5 Sonnet (Oct 2024) Anthropic | 51.6% | — | 22 Oct 2024 | — | Source ↗ | |
| 33 | Grok-3 mini xAI | 49.3% | high | 24 Jun 2025 | — | Source ↗ | |
| 34 | DeepSeek V3 Open source DeepSeek | 48.4% | — | 26 Dec 2024 | — | Source ↗ | |
| 35 | Gemini 2.5 Flash Preview (Apr 2025) Google DeepMind | 47.1% | — | 17 Apr 2025 | — | Source ↗ | |
| 36 | GPT-4o (Mar 2025) OpenAI | 45.3% | — | 27 Mar 2025 | — | Source ↗ | |
| 37 | GPT-4.5 Preview (Feb 2025) OpenAI | 44.9% | — | 27 Feb 2025 | — | Source ↗ | |
| 38 | gpt-oss-120b Open source OpenAI | 41.8% | high | 5 Aug 2025 | — | Source ↗ | |
| 39 | Qwen3 32B Open source Qwen | 40.0% | — | 29 Apr 2025 | — | Source ↗ | |
| 40 | Gemini 2.0 Flash (Dec 2024) Google DeepMind,Google | 38.2% | — | 11 Dec 2024 | — | Source ↗ | |
| 41 | Gemini 2.0 Pro Exp (Feb 2025) Google DeepMind | 35.6% | — | 5 Feb 2025 | — | Source ↗ | |
| 42 | o1-mini (unknown thinking) OpenAI | 32.9% | unknown | 12 Sep 2024 | — | Source ↗ | |
| 43 | GPT-4.1 Mini OpenAI | 32.4% | — | 14 Apr 2025 | — | Source ↗ | |
| 44 | Claude 3.5 Haiku (Oct 2024) Anthropic | 28.0% | — | 22 Oct 2024 | — | Source ↗ | |
| 45 | GPT-4o (Jan 2025) OpenAI | 27.1% | — | 29 Jan 2025 | — | Source ↗ | |
| 46 | GPT-4o (Aug 2024) OpenAI | 23.1% | — | 6 Aug 2024 | — | Source ↗ | |
| 47 | Qwen2.5-Max Alibaba | 21.8% | — | 25 Jan 2025 | — | Source ↗ | |
| 48 | QwQ-32B Open source Alibaba | 20.9% | — | 5 Mar 2025 | — | Source ↗ | |
| 49 | Gemini 2.0 Flash Thinking Exp Google DeepMind,Google | 18.2% | — | 21 Jan 2025 | — | Source ↗ | |
| 50 | GPT-4o (Nov 2024) OpenAI | 18.2% | — | 20 Nov 2024 | — | Source ↗ | |
| 51 | DeepSeek-V2.5 (Sep 2024) Open source DeepSeek | 17.8% | — | 6 Sep 2024 | — | Source ↗ | |
| 52 | Qwen2.5 Coder 32B Instruct Open source qwen | 16.4% | — | 11 Nov 2024 | — | Source ↗ | |
| 53 | Llama 4 Maverick Open source Meta | 15.6% | — | 6 Apr 2025 | — | Source ↗ | |
| 54 | Yi-Lightning 01.AI | 12.9% | — | 2 Dec 2024 | — | Source ↗ | |
| 55 | Cohere Command A Open source Cohere | 12.0% | — | 13 Mar 2025 | — | Source ↗ | |
| 56 | Codestral Open source Mistral AI | 11.1% | — | 13 Jan 2025 | — | Source ↗ | |
| 57 | openhands-lm-32b-v0.1 | 10.2% | — | 26 Mar 2024 | — | Source ↗ | |
| 58 | GPT-4.1 Nano OpenAI | 8.9% | — | 14 Apr 2025 | — | Source ↗ | |
| 59 | Gemma 3 27B Open source Google | 4.9% | — | 12 Mar 2025 | — | Source ↗ | |
| 60 | GPT-4o-mini OpenAI | 3.6% | — | 18 Jul 2024 | — | Source ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.