State of Play: AI Coding Assistants - InfoQ

Birgitta Böckeler, Global Lead for AI-assisted Software Delivery at Thoughtworks, delivered a comprehensive presentation at QCon London examining the rapid evolution of AI coding assistants over the past year. Her talk, titled "State of Play: AI Coding Assistants," provides valuable insights into how these tools have matured from simple autocomplete features to sophisticated autonomous agents capable of complex software development tasks.

The Evolution of Context Engineering

One of the most significant developments in AI coding assistants has been the evolution of context engineering. A year ago, context engineering primarily involved rules files like AGENTS.md or CLAUDE.md that developers would place in their workspaces. These files contained instructions about coding conventions, common pitfalls, and repeated errors that agents frequently made.

Since then, the landscape has expanded dramatically. Böckeler highlighted several new approaches:

• Skills: Introduced by Anthropic, skills allow developers to modularize rules into subfolders that can be loaded just-in-time by the LLM. This progressive loading approach prevents context window overload while providing relevant information exactly when needed.
• Commands and Subagents: Modern coding assistants can spawn subagents for specific tasks like code exploration or review, keeping the main session focused and efficient.
• CLI Integration: Many developers are shifting away from MCP servers toward using existing CLI tools and scripts, finding this approach more straightforward and reliable.

Böckeler emphasized that context engineering is essentially about amplifying both good and bad practices. Developers must carefully consider what coding conventions to reinforce, what workflows to build for modernization initiatives, and what tools to make available to agents.

The Push Toward Agent Autonomy

The trend toward giving agents more autonomy has accelerated significantly. Böckeler identified several manifestations of this trend:

1. Cloud-based agents: Tools like Claude Code, Cursor, and GitHub Copilot now offer cloud-based versions that can work unsupervised for extended periods.
2. Headless CLI modes: Command-line versions of coding assistants can be integrated into existing CI/CD pipelines and automation workflows.
3. Agent swarms: Experimental approaches where dozens or hundreds of agents work in parallel on problems, with coordination mechanisms to manage their interactions.

However, Böckeler cautioned against the hype surrounding agent swarms. She noted that successful experiments like Cursor's browser-building project and Anthropic's C compiler project worked because they tackled well-specified problems with abundant online documentation and test suites—conditions rarely found in enterprise software development.

Security and Cost Concerns

As agents gain more autonomy, two critical concerns have emerged: security and cost.

Security Risks

The security landscape has become increasingly complex:

• Prompt injection: Agents can be manipulated through untrusted content to execute unwanted commands or extract secrets.
• The "lethal trifecta": When agents have exposure to untrusted content, access to private data, and external communication capabilities, the risk of data breaches and security incidents increases dramatically.
• Sandboxing challenges: Even locally running agents need proper sandboxing to prevent unauthorized access to system resources.

Böckeler referenced Simon Willison's framework for understanding these risks, emphasizing that business use cases involving email integration or other external communications are particularly vulnerable.

Cost Explosion

The honeymoon period for AI coding assistants appears to be over. What once cost "12 cents per 100 lines of code" has ballooned significantly:

• Daily token usage of $380 has been reported by individual developers
• Monthly costs can reach levels comparable to developer salaries
• The shift from simple autocomplete to comprehensive agent workflows has dramatically increased token consumption

Böckeler explained that modern workflows involve extensive research, planning, implementation, testing, and review phases, each consuming significant tokens even when the final output is minimal.

Building Trust Through Harness Engineering

To address concerns about maintainability and reliability, Böckeler introduced the concept of "harness engineering"—creating safety nets around AI agents to increase trust in their outputs.

Forward-Feed Harness

This involves anticipating potential issues and providing agents with:

• Principles and conventions: Clear guidelines for coding standards and architectural decisions
• Reference documentation: How-to guides and best practices
• Tool access: CLIs, Codemods, language servers, and other deterministic tools
• Language servers: Tools like IntelliJ's refactoring capabilities that provide structured code understanding

Feedback Harness

After initial code generation, agents can use:

• Static analysis tools: Linters and structural tests that provide immediate feedback
• Custom linters: Enhanced error messages that guide agents toward better solutions
• Test suites: Both generated and deterministic tests to verify functionality

Böckeler experimented with dependency-cruiser for TypeScript projects, setting up rules about module imports and architectural constraints. She found that enhancing linter messages with agent-specific guidance—essentially "good type of prompt injection"—helped agents understand and correct issues more effectively.

The Goldilocks Speed Problem

While the push for speed remains strong, Böckeler questioned whether faster is always better. She introduced the concept of "Goldilocks speed"—fast enough to be productive but not so fast that it compromises quality and maintainability.

Key considerations include:

• Risk assessment: Evaluating probability, impact, and detectability for each task
• Experience requirements: More autonomous workflows require more experienced developers who can handle cognitive load
• Organizational pressure: The demand for speed can lead to corner-cutting and sloppy practices
• Maintainability trade-offs: Rapid development may create technical debt that becomes problematic over time

Böckeler cited Amazon's response to AI-related outages—adding more review gates—as an example of how the pursuit of speed can paradoxically slow things down.

Looking Forward: Questions and Considerations

Böckeler concluded by posing several critical questions for organizations considering increased AI autonomy:

1. Readiness assessment: How prepared is your organization to give AI more autonomy?
2. Safety net evaluation: What automated safety measures do you have in place?
3. Security posture: How robust is your security stance for AI-assisted development?
4. AI literacy: What is the current level of AI understanding among your engineering team?

She emphasized that improving these areas benefits both AI-assisted and traditional development practices. Organizations can use AI itself to help build and improve their safety nets, creating a virtuous cycle of improvement.

The Future Landscape

The presentation painted a picture of AI coding assistants evolving into sophisticated tools that require careful management and governance. While the technology continues to advance rapidly, the human element—experienced developers making informed decisions about when and how to use these tools—remains crucial.

Böckeler's analysis suggests that the future of software development will likely involve a spectrum of AI assistance, from simple autocomplete to fully autonomous agents, with developers choosing the appropriate level based on task complexity, risk tolerance, and organizational maturity. The key will be finding the right balance between automation and human oversight, speed and quality, innovation and stability.

The evolution from "vibe coding" to sophisticated harness engineering represents a maturation of the field, acknowledging that while AI can dramatically accelerate development, it requires thoughtful implementation to deliver sustainable value.