Why the Open‑Online CTF Format Is Losing Its Meaningful Competitive Edge

What the community is claiming

• The format is dead. Veteran players argue that open‑online CTFs no longer reward human skill because large language models (LLMs) can solve most challenges with minimal human input.
• Scoreboards are corrupted. Rankings on sites like CTFTime now reflect how many tokens a team can spend on AI agents rather than how deep their exploit knowledge is.
• Learning pipelines are broken. Beginners lose the incremental feedback loop that used to let them climb a visible ladder; the top of the leaderboard is increasingly occupied by AI‑orchestrated teams.

What actually changed

Year	Key development	Effect on CTFs
2023	Release of GPT‑4 (OpenAI)	Medium‑difficulty crypto and reverse‑engineering puzzles became solvable with a single prompt. Human effort saved, but hard problems still required manual work.
2024	Claude Opus 4.5 and Claude Code CLI	Fully scripted pipelines could query the model for each challenge via the CTFd API, automatically submitting flags after the first hour. Teams that integrated the pipeline outpaced those that relied on manual analysis.
2025	GPT‑5.5 / GPT‑5.5 Pro (OpenAI)	Benchmarks show parity with Anthropic’s Mythos series. The models can generate working exploit code for “Insane” heap‑pwn tasks on platforms like HackTheBox. In a 48‑hour event, a well‑orchestrated agent often clears the board before human teams finish.
2026	Specialized security LLMs (e.g., alias1 by Alias Robotics) lose relevance	General‑purpose frontier models dominate because they already contain the necessary reasoning and code‑generation capabilities, making niche models unnecessary.

The technical shift is not that tools are allowed—players have always used debuggers, fuzzers, and scripts. The shift is that the reasoning step is outsourced to a model that can read a challenge description, synthesize an attack, and output a flag without the human needing to understand the underlying vulnerability.

Why the impact is larger than “just AI assistance”

1. Orchestration becomes the competition. The bottleneck moves from finding a flaw to building a reliable automation pipeline (API wrappers, token budgeting, result aggregation). This favors teams with engineering resources rather than pure exploit skill.
2. Scoreboard distortion. Historically, a team’s rise on the leaderboard signaled growing expertise. When a bot can sweep medium challenges in minutes, the leaderboard no longer mirrors human learning curves, discouraging newcomers who see no visible progress.
3. Challenge author fatigue. Designing a multi‑hour puzzle that survives a Claude‑4‑style model requires deliberately anti‑LLM tricks (e.g., obfuscating semantics, requiring interactive binary debugging). Many authors find this effort unrewarding, leading to fewer high‑quality challenges being released.
4. Economic imbalance. Token costs are a real expense; teams with larger budgets can afford to run many parallel agents, effectively turning the competition into a pay‑to‑win race.

Limitations of the current AI‑driven state

• Hardest finals still resist automation. The final rounds of elite events like DEF CON’s CTF still require deep, novel reasoning that current models struggle with. However, these finals are reached only after qualifying rounds that many AI‑orchestrated teams have already cleared, reducing the pool of human participants.
• Model hallucinations. Even top‑tier LLMs produce incorrect exploit code or malformed flags at non‑trivial rates. Human oversight is still needed to verify and correct outputs, especially for “Insane” binary exploitation.
• Training‑cutoff leakage. Models trained on public repos and past CTF write‑ups can inadvertently contain solutions to older challenges, but they do not magically understand brand‑new, unpublished techniques.
• Regulatory enforcement is weak. Rules that forbid LLM use are hard to police in open‑online events; the only practical deterrent is community stigma, which is insufficient when the competitive advantage is large.

What this means for the CTF ecosystem

1. The ladder effect is eroding. Beginners can no longer rely on a clear, merit‑based progression path. The visible metric (leaderboard rank) now reflects token spending and orchestration skill, not the mastery of reverse‑engineering or cryptanalysis.
2. Recruitment pipelines are weakened. Companies that used CTF performance as a proxy for security talent must now supplement with other assessments (e.g., live coding interviews, bug‑bounty track records).
3. Community cohesion is at risk. As top teams drop out or shift focus to AI‑centric pipelines, the social glue—team chats, post‑CTF debriefs, shared write‑ups—diminishes.

Possible ways forward

• Separate AI‑assisted tracks. Organisers could run parallel divisions: one that explicitly allows LLM orchestration (scoring automation efficiency) and another that bans model access, preserving a human‑skill leaderboard.
• Shift emphasis to education platforms. Environments like picoGym or Hack The Box already treat the experience as a learning journey rather than a competitive ranking. Encouraging newcomers to start there may retain the educational value.
• Introduce “human‑only” challenges. Design puzzles that require interactive debugging, side‑channel analysis, or physical‑hardware interaction—tasks that are currently impractical for pure LLM pipelines.
• Transparent token‑budget scoring. Publish the amount of compute or tokens spent per flag; this would make the cost of AI assistance visible and could be factored into rankings.

Bottom line

The claim that “CTFs are dead” is an overstatement, but the underlying observation is solid: the open‑online format has been transformed from a skill‑based competition into an AI‑orchestration benchmark. The community’s challenge now is to preserve the educational ladder and the craft of challenge design while acknowledging that frontier LLMs are an unavoidable part of the modern security toolbox. Without deliberate structural changes, the visible scoreboard will continue to diverge from human expertise, and the very incentive that once drove countless students into security research may fade.

For further reading on LLM‑driven security tooling, see the recent analysis of Claude Code’s CLI integration on the Anthropic blog and OpenAI’s technical report on GPT‑5.5 capabilities.