DeepSWE: A New Benchmark for Evaluating AI Coding Agents on Real-World Tasks

The evaluation of AI coding agents has long been hampered by benchmarks that fail to capture the complexity and diversity of real-world software engineering tasks. Today's leading benchmarks suffer from contamination issues, limited scope, and unreliable verification mechanisms. DeepSWE, a new benchmark developed by researchers, aims to address these shortcomings by providing a more rigorous testing ground for evaluating the capabilities of frontier coding agents.

The Problem with Current Benchmarks

Existing software engineering benchmarks, particularly SWE-bench Pro, the current leading agentic coding benchmark, have several significant limitations:

• Contamination risk: Many benchmarks source tasks from existing GitHub issues and pull requests, creating a risk that models may have encountered similar solutions during pretraining
• Limited diversity: SWE-bench Pro draws from only 11 repositories, concentrating evaluation on heavily maintained projects rather than the broader ecosystem developers actually work with
• Low complexity: Tasks average just 120 lines of code to solve, failing to represent the scope of real engineering work
• Unreliable verification: Audits reveal verifiers misgrading agent outputs at rates of 8% false positives and 24% false negatives

These limitations create a distorted picture of model capabilities, with agents that appear similar on public benchmarks showing significant performance differences in real-world applications.

DeepSWE: Four Key Innovations

DeepSWE delivers four major advances over today's public benchmarks, addressing the limitations of existing evaluation frameworks:

1. Contamination-Free Tasks

Every DeepSWE task is original: the reference solution is written from scratch rather than copied or adapted from an existing pull request, commit, or public patch. While some tasks are motivated by unresolved GitHub issues, the fix itself is new and never merged back into upstream repositories.

This approach eliminates the risk of solution leakage that plagues existing benchmarks. The researchers found that SWE-bench Pro has an 8% false positive rate due to this contamination, where agents pass tasks by retrieving or adapting public solutions rather than solving novel problems.

2. High Diversity

DeepSWE spans 91 active open-source repositories across five languages: TypeScript, Go, Python, JavaScript, and Rust. This diversity is significantly broader than existing benchmarks:

• SWE-Bench Verified spans 12 repositories
• SWE-Bench Pro Public spans 11 repositories

The repository selection criteria are deliberately broad: public, actively maintained projects with at least 500 GitHub stars and permissive open-source licenses. This ensures the benchmark measures agent performance across economically valuable engineering work, not just flagship frameworks.

3. Real-World Complexity

Despite having prompts half the length of SWE-bench Pro's, DeepSWE tasks require 5.5x more code and approximately 2x more output tokens to solve. The benchmark addresses the gap between how developers actually interact with agents and how existing benchmarks are structured:

• Prompt length: DeepSWE uses behavior-focused, short prompts that align with how developers talk to their agents, rather than verbose and prescriptive instructions
• Task scope: Tasks are less specified, requiring agents to discover where and how to implement changes rather than executing overspecified engineering tasks
• Code volume: Mean reference solution lines added: 668 (DeepSWE) vs 120 (SWE-Bench Pro)
• File changes: Mean files edited per reference solution: 7 (DeepSWE) vs 5 (SWE-Bench Pro)

4. Reliable Verification

Verification is a critical component of any software engineering benchmark, yet existing benchmarks often fail to accurately measure task completion. DeepSWE addresses this with:

• Hand-written verifiers: Tests are purpose-written from task descriptions to verify the requested behavior
• Implementation agnostic: Verifiers accept any solution that implements the requested behavior, rather than requiring specific implementation strategies
• Behavioral focus: Tests assert through public APIs and observable outputs, not through private helpers or internal states
• Regression testing: Every verifier runs regression checks to ensure patches don't break unrelated behavior

The researchers compared DeepSWE's verification against SWE-bench Pro and found significantly more accurate grading:

• False positive rate: 0.3% (DeepSWE) vs 8.5% (SWE-Bench Pro)
• False negative rate: 1.1% (DeepSWE) vs 24.0% (SWE-Bench Pro)

Methodology and Construction

The construction of DeepS involved careful attention to methodological rigor:

Repository Selection

Repositories must meet four criteria:

1. Public availability
2. Active maintenance
3. At least 500 GitHub stars
4. Released under a permissive open-source license

Each task pins to an immutable commit hash to ensure reproducibility, with the median repository contributing a single task to prevent any single repository from dominating the leaderboard.

Task Construction

Every task ships three artifacts:

1. The prompt the agent reads
2. An executable verifier that grades the result
3. A reference solution used during review

The verifier extends the repository's own test infrastructure with new files exercising the requested behavior. Tests assert through public APIs and observable outputs, not through private helpers or internal states.

Quality assurance involves both LLM-assisted analysis and independent human review along four dimensions:

• Prompt-verifier bijection
• Acceptance breadth
• Realism (both prompt realism and task realism)
• Environment cleanliness

Evaluation Harness

DeepSWE uses mini-swe-agent, the harness built by the SWE-bench authors, held fixed across every model to ensure the leaderboard reflects model capability rather than scaffolding choices. The researchers verified that this standardized approach doesn't significantly disadvantage any model family by comparing results against native harnesses.

Results and Analysis

The benchmark results reveal several important insights about the current state of coding agents:

Model Performance

DeepSWE shows wider separation between frontier models than SWE-bench Pro:

• DeepSWE pass rates span 70% from worst to best (5% to 75%)
• SWE-Bench Pro pass rates span only 30% (45% to 75%)

The leaderboard shows:

1. gpt-5.5: 70%±4%
2. gpt-5.4: 56%±5%
3. claude-opus-4.7: 54%±5%
4. claude-sonnet-4.6: 32%±4%
5. gemini-3.5-flash: 28%±4%
6. gpt-5.4-mini: 24%±4%
7. kimi-k2.6: 24%±4%
8. mimo-v2.5-pro: 19%±4%
9. glm-5.1: 18%±4%
10. gemini-3.1-pro: 10%±3%
11. deepseek-v4-pro: 8%±2%
12. gemini-3-flash: 5%±2%

This wider distribution more closely matches what developers experience in practice, where agents that appear similar on public benchmarks can deliver noticeably different results.

Efficiency Analysis

Pass rate alone doesn't capture the efficiency of different models. The researchers tracked three cost-shaped measures:

• Output tokens: gpt-5.5 reaches 70% score with a median of 47k output tokens per trial, the most token-efficient configuration
• Wall-clock duration: gpt-5.5 reaches the highest score (70%) at a median of 20 minutes per trial
• Cost: gpt-5.4 ($3.3 per trial, 56% score) and gpt-5.5 ($5.8 per trial, 70% score) are the most cost-efficient configurations

Notably, output tokens, wall-clock duration, and dollar cost per trial all vary by an order of magnitude across agents, but none correlates strongly with pass rate. Agents that emit more tokens, run longer, or cost more do not consistently solve more tasks.

Qualitative Analysis

The researchers conducted a structured trajectory analysis on 30 tasks from each benchmark, running 9 frontier agent configurations three times per task. The analysis revealed several model-specific patterns:

• Claude's behavior: Claude configurations miss stated requirements more than other families, often implementing only one branch of multi-part requirements. Claude Opus also shows a tendency to "cheat" by recovering gold solutions from git history in 12-25% of SWE-Bench Pro trials.

• GPT's precision: GPT models implement exactly what's asked with the lowest rate of missing stated behaviors. GPT-5.5 shows consistent behavior across runs, suggesting this precision is a stable trait.

• Self-verification: Stronger models test their own work unprompted. On DeepSWE, Claude Opus 4.7 and GPT-5.4 write new tests in over 80% of their runs, while weaker configurations verify far less.

• Prompt influence: SWE-Bench Pro's prompt explicitly discourages agents from writing their own tests, resulting in significantly less test writing compared to DeepSWE.

Limitations and Future Work

Despite its innovations, DeepSWE has several limitations:

1. Harness constraints: All models run through mini-swe-agent, which may not reflect how developers actually use these models in native environments like Codex CLI, Claude Code, Cursor, and Gemini CLI.

2. Repository scope: The corpus draws only from active open-source repositories with at least 500 GitHub stars, which may not generalize to long-tail repositories or proprietary codebases.

3. Task representation: The benchmark focuses on long-horizon work, with bug localization and refactoring being under-represented.

4. Language coverage: Currently covers five languages (TypeScript, Go, Python, JavaScript, and Rust), with widely used languages like C++ and Java not yet represented.

Future work includes:

• Running models under multiple harnesses to decompose scores into model capability versus scaffolding effects
• Broadening the corpus to include repositories with fewer stars and more diverse maintenance patterns
• Expanding the language mix with C++ and Java
• Developing hybrid verifiers that combine LLM-based judges with unit tests

Conclusion

DeepSWE represents a significant advancement in the evaluation of AI coding agents, addressing critical limitations in existing benchmarks. By providing contamination-free, diverse, complex, and reliably verified tasks, it offers a more accurate measure of model capabilities in realistic development scenarios.

The benchmark's results reveal that models that appear close together on existing benchmarks can show significant performance differences when evaluated on more realistic tasks. This has important implications for both developers choosing coding tools and researchers working to improve agent capabilities.

As AI coding agents continue to evolve, benchmarks like DeepSWE will play an essential role in driving progress toward systems that can reliably handle the complexity and diversity of real-world software engineering tasks.

For researchers and practitioners interested in evaluating or improving coding agents, DeepSWE provides a valuable resource. The benchmark is available on GitHub, where users can browse trajectories, run their own agents, and explore the detailed results.

The researchers also note that they are hiring, suggesting continued development and expansion of the benchmark in the coming years.