AI agents still need human teachers, study finds

A comprehensive new study has found that AI agents still fundamentally rely on human expertise to perform complex tasks effectively, despite growing expectations that these systems might eventually teach themselves.

The research, conducted by a team of 40 computer scientists from major tech companies and universities including Amazon, Stanford, UC Berkeley, and Carnegie Mellon, examined how AI agents perform when given different types of "skills" - essentially reference materials and instructions that augment their capabilities beyond what was captured in their original training data.

The experiment design

The researchers tested seven different agent-model configurations across 84 distinct tasks, generating 7,308 individual attempts to solve problems. They compared three conditions: agents with no additional skills, agents with human-curated skills, and agents asked to generate their own skills.

The tasks ranged from technical challenges like flood-risk analysis and software engineering to domain-specific work in healthcare and manufacturing. For each task, the agents attempted to complete the work using their base capabilities, then with curated human guidance, and finally by trying to teach themselves the necessary skills.

Human-curated skills deliver dramatic improvements

When equipped with human-designed skills, AI agents showed substantial performance gains. On average, agents with curated skills completed tasks 16.2 percent more frequently than those without any additional guidance.

One striking example involved flood-risk analysis. Agents without skills achieved only a 2.9 percent success rate because they failed to apply the appropriate statistical methods. However, when given a curated skill that specified using the Pearson type III probability distribution, applying USGS methodology, and including specific code examples with scipy function calls, the success rate jumped to 80 percent.

Domain expertise matters most where training data falls short

The study revealed that human-curated skills provided the greatest benefit in domains where specialized knowledge is typically underrepresented in training data. Healthcare skills improved performance by 51.9 percentage points, while manufacturing skills boosted results by 41.9 percentage points.

In contrast, domains like mathematics and software engineering saw smaller gains of 6.0 and 4.5 percentage points respectively, suggesting these areas are better represented in the agents' original training.

Less is more when it comes to skill design

Interestingly, the researchers found that skill modules work best when kept concise. Skills containing only 2-3 focused modules outperformed massive data dumps, suggesting that targeted, well-organized guidance is more effective than overwhelming agents with information.

This principle extended to model scale as well. Smaller models like Anthropic's Claude Haiku 4.5, when equipped with curated skills, actually outperformed larger models like Claude Opus 4.5 that lacked such guidance. Haiku with skills achieved a 27.7 percent success rate compared to Opus without skills at 22 percent.

Self-generated skills fail to deliver

The most surprising finding came when agents were asked to generate their own skills. The researchers instructed agents to analyze task requirements, identify necessary domain knowledge and APIs, write 1-5 modular skill documents, save them as markdown files, and then attempt the task using their self-generated reference material.

The results were disappointing. Agents using self-generated skills performed worse than those with no skills at all, showing a negative 1.3 percentage point average impact on task completion.

"Self-generated skills provide negligible or negative benefit, demonstrating that effective skills require human-curated domain expertise," the authors concluded.

Implications for AI development

These findings challenge the narrative that AI systems are rapidly approaching autonomous learning capabilities. While machine learning models can improve through training on large datasets, the study shows that during actual use - the inference phase - these agents still fundamentally depend on human guidance for complex, specialized tasks.

The proliferation of skill directories and the rapid development of agent capabilities might suggest an ecosystem moving toward self-sufficiency, but this research indicates that human expertise remains essential for bridging the gap between general training and specific task requirements.

The continuing role of human teachers

For now, the AI revolution remains a collaborative effort between human teachers and their machine students. The study demonstrates that while AI agents have become remarkably capable tools, they still need human curators to provide the specialized knowledge and structured guidance necessary for high-stakes, domain-specific work.

As AI agents continue to proliferate across industries, from Claude Code to Gemini CLI to Codex CLI, this research suggests that investment in human expertise and careful skill design will remain critical for achieving reliable, high-performance results in real-world applications.

The full study, titled "SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks," provides a framework for evaluating and improving how we augment AI agents, ensuring that the human element in artificial intelligence remains strong even as the technology continues to advance.