Rust Project Grapples with AI: A Spectrum of Perspectives on Technology's Impact on Programming and Open Source

The Rust programming language project, known for its commitment to performance, safety, and developer experience, has undertaken a significant exercise in understanding how artificial intelligence intersects with its community and development practices. Beginning in February 2023, the project initiated a collection of perspectives on AI, culminating in a comprehensive summary document authored by nikomatsakis that captures the breadth of opinion within the Rust ecosystem.

This document represents an important snapshot of how a mature, influential open-source project is grappling with one of technology's most transformative forces. Rather than presenting a unified stance, the compilation reveals a spectrum of viewpoints that reflect the broader tech community's ambivalence toward AI tools in software development.

AI as a Sophisticated Tool

A recurring theme across many contributions is the recognition that AI, particularly large language models, is not a simple plug-and-play solution but rather a sophisticated tool requiring significant engineering expertise to use effectively. As one contributor noted, "It takes care and careful engineering to produce good results. One must work to keep the models within the flight envelope."

This perspective helps explain the widely divergent experiences developers report with AI tools. Some find them transformative, while others find them frustratingly unhelpful. The difference often appears to lie not in the tools themselves, but in how they're applied. "Something that might not be obvious is how much things have changed over the last 2-3 months," observed one contributor, highlighting the rapid evolution of AI capabilities that has made recent models "too good to ignore."

The distinction between expert and novice usage was frequently mentioned, with several contributors noting that AI can amplify capabilities in experienced hands while potentially hindering skill development in those still learning their craft. This creates a complex dynamic where the same technology can simultaneously empower established developers and complicate the journey for newcomers.

Beyond Code: AI's Value in Other Development Tasks

While much of the discussion focused on AI for code generation, many contributors highlighted valuable applications in other areas of development. Several mentioned AI's effectiveness in research and discovery, particularly when navigating unfamiliar codebases or documentation.

One contributor shared their experience with internal AI tooling at Arm: "We have some internal AI tooling at Arm that makes searching our 10,000+ page architecture documentation much easier, and I find that exceptionally valuable - it makes a lot easier for me to respond to issues upstream promptly."

Others found AI useful for "rubberducking" or brainstorming ideas, and some explored its potential in code review processes. One contributor noted, "I would be interested in exploring LLMs for code review. Some Linux kernel folks apparently had good success on having LLM agents assist in review using very project-specific, carefully crafted prompts."

Perhaps most notably, AI demonstrated value in handling semi-structured data processing tasks that would otherwise be prohibitively time-consuming. One example involved migrating glossary content into documentation chapters: "Once the directive format was nailed down, the agent was able to progressively, bit by bit, migrate all the glossary into the chapters to have a single source of truth from which the glossary can be generated wholesale."

Despite these successes, contributors frequently noted limitations, particularly in AI's ability to generate coherent, structured writing beyond the sentence level. "At the sentence level it was very good, at the paragraph level it was good, and at levels beyond that it was terrible," one contributor observed of an AI-generated project documentation. "Bad structure, repetitive, no sense of order or flow. Just feels like a random collection of related things."

Diverse Coding Experiences

Experiences with AI for coding tasks varied dramatically among contributors. Some found it inefficient: "It takes more time for me to coerce AI tooling to produce the code I want plus reviews and fixes, than it is for me to just write the code myself," stated one developer.

Others, however, described transformative experiences. "If I had to pick one word for how I feel about using AI, it is empowered," shared nikomatsakis. "Suddenly it feels like I can take on just about any problem – it's not that the AI will do the work for me. It's that the AI will help me work through it and also tackle some of the drudgery."

Several contributors noted that AI seemed particularly effective for well-constrained tasks or specific domains. "I've enjoyed LLMs for writing proc macro code because that's no fun and not too correctness-sensitive," one maintainer observed. "They've become quite good at writing Rust code lately."

This variability in effectiveness suggests that AI may complement certain programming activities better than others, a realization that could help developers more strategically incorporate these tools into their workflows.

The Atrophy Risk and Mental Model Building

A significant concern raised by multiple contributors was the potential for over-reliance on AI to lead to skill atrophy or diminished understanding of codebases. "It's really difficult to retain 'deep impressions' or develop mental models of the codebase for code that I didn't write myself," one developer noted.

This concern connected to broader theories about programming as a knowledge-building process. Nicholas Nethercote referenced Peter Naur's essay "Programming as Theory Building," which argues that "a program exists not just as source code, but also mental models in programmer's brains, and that the mental models are as important, or even more important, than the source code."

The question of what happens when these mental model-building processes are outsourced to AI was left largely unanswered but deeply concerning to many. "So what does it mean to outsource all of that to an LLM? I can't see it having a good outcome," Nethercote concluded.

Reviewing AI-Generated Code: A Unique Challenge

Even when AI produces technically correct code, reviewing it presents unique challenges. Several contributors noted that AI-generated code can contain subtle errors that are difficult to detect without deep understanding of the original problem.

"I think 'human reviews the resulting code thoroughly' works," one contributor stated. "Experimenting with the inline snippet in VSCode while working on the search graph got it to propose incorrect, but seemingly reasonable, comments. I even used some of them without realizing they are wrong."

Others pointed out that AI-generated code changes the fundamental nature of what code review is for. "Code reviews are not suited for catching minutia and are instead generally focused on reducing the bus factor by keeping other people abreast of changes," one maintainer observed. "But minutia reviews is what AI needs and the AI-using contributor is no longer an 'author' but a 'reviewer'."

This transformation of the review process could have significant implications for team dynamics and knowledge sharing within open-source projects.

The Learning Dilemma

Several contributors raised concerns about how AI might affect learning and skill development, particularly for newcomers to programming. "LLMs can be great tools in the hands of experts, but using them too much too early can prevent a person from even becoming an expert," warned RalfJung.

This concern was supported by references to research suggesting that AI assistance can hinder learning capabilities even while users believe they're learning effectively. "The science again and again points to either it being net negative in time spent, or to learning capabilities being hindered, all while participants believe they were faster or learned well respectively," one contributor noted.

Others approached this from a "choice architecture" perspective, suggesting that making AI tools easily accessible might create incentives for taking shortcuts that undermine deep learning. "If we view it from the choice architecture angle, allowing or enabling AI use risks making low effort & low engagement the most convenient," observed fee1-dead.

The Ethical Dimensions

Beyond practical considerations, many contributors raised profound ethical concerns about AI technologies. The provenance of training data emerged as a fundamental issue for many. "LLMs are trained on stolen data," stated one contributor bluntly. "It seems to me that, given the amount of data needed to train an LLM, it would not be possible to train one comparable to current models on licensed data."

Access inequalities were another concern, with some noting that the best AI tools are increasingly expensive and controlled by a small number of companies. "It is also a new kind of closed gardens where the best available LLMs are owned by few companies which keep increasing their prices while reducing the free 'trial', increasing yet again the gap these who can afford it, and the others," explained Guillaume Gomez.

Environmental impact also featured prominently in the discussion. "The energy demands result in increasing rather than reducing the emission of greenhouse gasses into the atmosphere," one contributor stated. "Coal plants that were slated to be closed are being kept alive. Large tech companies walking back their de-carbonisation commitments."

Several contributors also expressed concern about how AI might reinforce societal biases and concentrate power in the hands of a few corporations. "Furthermore, due to the costs of creating such models, there are only a few companies offering them, concentrating a lot of power and control of people's programming behavior in a few hands," RalfJung observed. "They are centralized, proprietary services which leave the vendor in full control over who uses them when and where to do what – the exact opposite of FOSS, of empowering individuals."

Legal Uncertainties

Adding to the complexity, the legal landscape surrounding AI remains unsettled. Contributors noted ongoing lawsuits in the US attempting to adjudicate fair use questions around AI training, the EU's emerging AI Act with its transparency requirements, and unresolved questions about copyright over AI-generated output.

For open-source projects that depend on being able to assert their license terms, these legal uncertainties present significant risks. "Unsolved copyright issues with no solution in sight," one contributor stated. "The current thinking is to assume good faith and hope for the best. For a FOSS project, this is a huge Sword of Damocles."

AI's Impact on Open Source

The effects of AI on open-source development processes emerged as a major theme. Several contributors noted that AI makes it easy to generate "plausible-looking" but incorrect contributions, creating challenges for maintainers.

"An official 'stamp of approval' can often be the missing impetus that enables many people, who previously might not have pumped out LLM slop as contributions, to do so with less guilt," one developer observed. "This of course doesn't represent all people, but it represents a (somewhat) growing majority of people."

The phenomenon of contributors using AI as a "proxy" to respond to reviewer comments was particularly frustrating. "A few contributors even act as a proxy between the reviewer and the LLM, copy their reviewer's question, reply with LLM-generated response," shared Jieyou Xu. "For the love of god, please. I want to emphasize this is incredibly frustrating. This is the top contributing factor to potential burn outs for me."

The volume of low-quality contributions was another concern. "We get way too many PRs 'fixing' things that don't actually solve it, and those seem to largely from AI and wouldn't have existed without it," stated scottmcm. "I continue to think that the greatest threat to the project is its lack of review bandwidth, and LLM is only making that worse, with no realistic prospect for it to make it better."

Beyond the code itself, several contributors argued that open-source projects involve more than just code artifacts. "An open source project is more than just a codebase. There is a community of people around it," Nicholas Nethercote explained. "These people have a shared commitment to the project. These people have a shared understanding of what the program does, and why."

From this perspective, AI-generated contributions that don't involve the human contributor in the community's knowledge-building processes represent a loss, even if technically correct.

Trust and Effort Signaling

A subtle but important concern raised was how AI breaks traditional signals of effort and commitment in open-source contributions. "My main concern is that LLMs break nearly all of our current ways to detect effort," lcnr observed. "This causes us to incorrectly allocate review and mentoring capacity."

In traditional open-source participation, preparing a reasonable-looking contribution required significant effort, which signaled the contributor's commitment and level of understanding. AI has decoupled this signal from the actual understanding and commitment, creating challenges for maintainers trying to assess where to focus their limited time and energy.

Navigating the Charged Atmosphere

The intensity of feeling around AI made productive conversation particularly challenging. On one side, embracing AI would alienate some contributors and users. "There will be a cost to accepting AI in terms of existing and potential contributors lost," noted oli-obk.

On the flip side, the charged atmosphere meant some people were reluctant to speak openly about their experiences. "How we present ourselves impacts what people will tell us, and we have some responsibility for that," nikomatsakis acknowledged. "I've gotten a number of messages from people expressing reluctance to talk about their use of AI publicly on Zulip."

Some contributors also noted that workplace pressures might be influencing public discourse. "Some folks have reached out privately to say that because of top-down pressure at their employer, they are reluctant to speak negatively about AI," nikomatsakis revealed.

Potential Paths Forward

Given these complex challenges, contributors proposed various potential approaches for the Rust project:

1. Universal Policies: Some suggested establishing clear policies around AI contributions, requiring contributors to understand and stand behind their work and explicitly disclose AI usage for substantial contributions.

2. Disclosure and Accountability: Requiring contributors to explicitly acknowledge that they have authored or reviewed their entire contribution themselves, with the ability to answer questions about it.

3. Language Accessibility: Encouraging people to write in their native language if not comfortable with English, rather than relying on AI translation.

4. Corporate Support: Seeking direct support from AI companies that use Rust, potentially through funding for maintainers.

5. Reputation Systems: Implementing trust-based systems or contribution thresholds to help reviewers prioritize higher-quality contributions.

6. AI-Assisted Moderation: Using AI tools to help identify low-quality contributions or assist with issue triage, though this approach raised its own set of concerns.

Common Ground and Core Tensions

Despite the disagreements, nikomatsakis identified several points of common ground:

• Maintainers are overburdened and that needs addressing
• Naive use of AI generates poor quality code
• Contributors must understand and stand behind their contributions
• Reviewers should be empowered to reject low-quality work
• AI-generated writing in issues and PR descriptions is particularly harmful
• AI has broken traditional signals of effort and commitment

However, several core tensions remain unresolved:

1. Deep Integration vs. Rejection on Moral Grounds: Some want enthusiastic embrace of AI, while others view any compromise as complicity in immoral practices.

2. "Supporting" AI vs. "Endorsing" AI: There's a question about whether improving documentation to help AI tools work better with Rust constitutes endorsement.

3. "Opting Out from AI" vs. "AI Can Help": The project needs to respect people's right to avoid AI while potentially leveraging it for tasks like translation or issue categorization.

The Path Ahead

The Rust project's compilation of perspectives serves as a valuable case study for how open-source communities might navigate the complex challenges and opportunities presented by AI technologies. Rather than arriving at definitive answers, the document highlights the need for ongoing, nuanced conversation that acknowledges both the potential benefits and significant risks these technologies present.

As nikomatsakis noted, the goal was not to present "the Rust project's view" but rather to capture the diversity of opinion to help understand "the landscape of opinion and the kinds of arguments on each side." This approach itself may offer a model for other communities grappling with AI's impact on collaborative knowledge work.

The Rust project's experience suggests that navigating AI's role in open-source development will require balancing technical considerations with ethical concerns, practical needs with philosophical principles, and individual autonomy with collective wellbeing. It's a challenge that extends beyond any single programming language or project, touching on fundamental questions about how we create and maintain knowledge in an age of increasingly sophisticated artificial intelligence.

In the end, the document stands as both a snapshot of current thinking and a starting point for the necessary ongoing conversations that will shape how AI and human collaboration evolve in the years to come.