AI Coding Agents: From Autocomplete to SWE-bench

From completion to repository-level work

The first generation of code models completed the next line. Codex (Chen et al., 2021) introduced the Codex family and HumanEval, a benchmark of short Python problems with unit tests. AlphaCode (Li et al., 2022) showed that a model could generate and filter candidate solutions for competitive programming. Code Llama and StarCoder widened the open-weight options.

Those systems were impressive, but the unit of work was still a function. A real engineering task starts with a repository, an issue, existing tests and conventions that are not written down. That is a different problem.

The benchmark shift

HumanEval and MBPP measure function-level synthesis. They are useful for comparing code models, but they do not measure whether an agent can navigate a repository, make a coherent change and keep the test suite green.

SWE-bench (Jimenez et al.) changed the evaluation target. Each task is a real GitHub issue paired with the repository state at the time and the tests that should pass after the fix. The agent must inspect the codebase, locate the relevant files, produce a patch and verify it. Success is measured by whether the tests pass, not by whether the patch looks plausible.

SWE-bench Verified, a human-validated subset released by OpenAI, reduced some of the ambiguity in the original benchmark. Even so, benchmark success is not the same as production readiness. The tasks are curated, the environment is controlled and the tests are known.

What made agents possible

Three changes mattered.

Context. Repository-level retrieval and tooling let the model inspect files, search symbols and read tests instead of relying on a single prompt.

Actions. Agents can run commands, apply patches, execute tests and iterate. SWE-agent (Yang et al.) showed that the interface between the model and the repository is a design variable: how files are opened, how edits are applied and how feedback is returned all affect performance. OpenHands (Wang et al.) packaged a broader agent runtime with tools, memory and evaluation.

Verification. A coding agent can check its own work by running tests, type checkers and linters. That feedback loop is what separates an agent from a code generator.

Verification is the product

The most reliable coding agents are not the ones that write the most code. They are the ones that verify the most. A small patch with passing tests beats a large patch that merely looks right.

A practical workflow:

1. Reproduce the issue with a failing test.
2. Make the smallest change that fixes it.
3. Run the relevant test subset.
4. Run the type checker and linter.
5. Review the diff for unintended changes.
6. Open a pull request with the trace and test results.

The agent should be able to explain why it changed each file. If it cannot, the patch is not ready for review.

Failure modes

Coding agents fail in predictable ways:

• Plausible but wrong patches that satisfy the visible test but not the intent.
• Test overfitting, where the agent edits tests instead of code.
• Context loss in large repositories.
• Unsafe commands executed without sandboxing.
• Prompt injection from repository content, issue text or dependency documentation.
• Silent dependency changes that break the build elsewhere.

Evaluate the trajectory, not only the final diff. Did the agent read the right files? Did it run the tests? Did it stop when the evidence was insufficient?

A workflow that works

Keep changes small. Give the agent a sandbox with no production credentials. Require tests for behavioural changes. Use CI as the final gate. Treat the agent as a fast junior engineer with good tools and no authority: useful, but never merged without review.

The benchmark race will continue. The engineering question is simpler: can the agent produce a verified, reviewable change without creating more work than it saves?

References

1. Chen et al. (2021). Evaluating Large Language Models Trained on Code.
2. Li et al. (2022). Competition-level Code Generation with AlphaCode. Science.
3. Rozière et al. (2023). Code Llama: Open Foundation Models for Code.
4. Li et al. (2023). StarCoder: May the Source Be with You!.
5. Austin et al. (2021). Program Synthesis with Large Language Models.
6. Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR.
7. OpenAI. Introducing SWE-bench Verified.
8. Yang et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS.
9. Wang et al. (2024). OpenHands: An Open Platform for AI Software Developers as Generalist Agents.
10. Greshake et al. (2023). Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec.