SkillsBench Reveals Agent Skills Are Hit-or-Miss: Some Tasks Soar, Others Crash

A new benchmark called SkillsBench has delivered a sobering reality check for the AI agent industry. While the concept of agent skills—structured packages of procedural knowledge that augment large language models at inference time—has been widely adopted, researchers have discovered that their effectiveness is far from guaranteed.

SkillsBench tested 7 agent-model configurations across 86 tasks spanning 11 domains, running over 7,300 evaluation trajectories. The results paint a nuanced picture: curated skills improved average pass rates by 16.2 percentage points, but the variation is staggering. Healthcare tasks saw a massive 51.9 percentage point boost, while software engineering tasks gained a modest 4.5 points. Even more concerning, 16 out of 84 tasks actually performed worse with curated skills.

"The data shows that skills aren't a universal solution," explains the research team led by Xiangyi Li. "Their impact depends heavily on the specific task and domain, and in some cases, they actively harm performance."

The most surprising finding? Models cannot reliably generate the procedural knowledge they benefit from consuming. When agents attempted to create their own skills, the results were flat—no average improvement whatsoever. This suggests a fundamental limitation in current AI systems' ability to introspect and optimize their own capabilities.

Interestingly, the research revealed that focused skills with just 2-3 modules often outperformed comprehensive documentation. This challenges the assumption that more information always leads to better outcomes. Additionally, smaller models equipped with well-curated skills could match the performance of larger models without any skills at all.

The benchmark introduces a standardized methodology for evaluating agent skills, pairing tasks with curated skills and deterministic verifiers. This addresses a critical gap in the field—until now, there's been no systematic way to measure whether these skills actually deliver on their promise.

For developers and organizations investing in AI agent infrastructure, the implications are clear: skills are a powerful tool, but they require careful curation and domain-specific optimization. The one-size-fits-all approach simply doesn't work. The research team has made their benchmark publicly available, providing a crucial resource for the community to evaluate and improve agent skill systems.

The findings suggest that the future of AI agents lies not in simply adding more skills, but in developing smarter methods for selecting, adapting, and applying the right skills to the right tasks. As the field matures, benchmarks like SkillsBench will be essential for separating genuine progress from marketing hype.