SWE‑fficiency: Measuring LLMs on Real‑World Python Performance Tuning

From Benchmarks to the Battlefield

In the world of large language models (LLMs), most performance evaluations are toy problems—sorting a list, translating a sentence, or answering a trivia question. SWE‑fficiency flips that script. It extracts real performance‑engineering pull requests from nine widely‑used Python libraries—

• numpy, pandas, scipy
• scikit‑learn, matplotlib
• xarray, sympy, dask, astropy

— and turns them into a reproducible, containerized testbed. Each task gives an agent a full repository snapshot, a workload script that measures runtime, and the existing unit‑test suite.

The Speedup Ratio: A Human‑Centric Metric

The core of SWE‑fficiency is the Speedup Ratio (SR):

SR = (Model Speedup) / (Expert Speedup)

An SR of 1.0× means the model matched the human engineer’s optimization. Surpassing 1.0× indicates the model not only caught up but exceeded human performance on that task. Expert speedups range wildly—from a modest 1.1× to a staggering 100×+—providing a rich spectrum of difficulty.

What the Numbers Say

When the benchmark was run against the latest frontier models—OpenHands and SWE‑agent scaffolds—results were sobering:

• The best models achieved < 0.15× the expert speedup on average.
• The gap widened on more complex, multi‑module tasks where models struggled to localize bottlenecks and reason about cross‑function execution.

This underperformance is not a simple lack of data; it reflects the investigative nature of performance engineering. Human experts spend time profiling, inspecting call chains, and iteratively validating hypotheses against targeted tests—steps that are hard for current LLMs to emulate.

Why It Matters

SWE‑fficiency is more than a benchmark; it is a diagnostic tool that pinpoints where LLMs falter:

1. Repository‑scale code understanding – parsing dozens of files, following imports, and grasping library internals.
2. Performance reasoning – identifying hot paths, estimating the impact of a change, and prioritizing fixes.
3. Correctness‑preserving edits – modifying code without breaking any unit test, even in the face of subtle edge cases.

These are the same skills that real engineers use when they tune a library that powers millions of data‑science pipelines. By exposing the gap, SWE‑fficiency forces the community to develop better tooling—static analysis, dynamic profiling, and test‑driven validation—within the LLM workflow.

The Road Ahead

The benchmark’s design already nudges research toward more realistic problem settings. Future iterations could add
- Hardware‑aware optimizations (GPU vs CPU, memory‑bandwidth constraints)
- Multi‑objective trade‑offs (speed vs. memory vs. numerical stability)
- Collaborative debugging where agents converse with human mentors.

For now, SWE‑fficiency stands as a clarion call: performance engineering is a complex, multi‑step process that LLMs have yet to master. Bridging that gap will unlock a new generation of AI‑assisted developers who can write faster code without sacrificing correctness.