RoguePilot Flaw in GitHub Codespaces Enabled Copilot to Leak GITHUB_TOKEN

A critical vulnerability in GitHub Codespaces could have been exploited by bad actors to seize control of repositories by injecting malicious Copilot instructions in a GitHub issue. The artificial intelligence (AI)-driven vulnerability has been codenamed RoguePilot by Orca Security. It has since been patched by Microsoft following responsible disclosure.

"Attackers can craft hidden instructions inside a GitHub issue that are automatically processed by GitHub Copilot, giving them silent control of the in-codespaces AI agent," security researcher Roi Nisimi said in a report.

The vulnerability has been described as a case of passive or indirect prompt injection where a malicious instruction is embedded within data or content that's processed by the large language model (LLM), causing it to produce unintended outputs or carry out arbitrary actions.

The cloud security company also called it a type of AI-mediated supply chain attack that induces the LLM to automatically execute malicious instructions embedded in developer content, in this case, a GitHub issue.

How the RoguePilot Attack Works

The attack begins with a malicious GitHub issue that then triggers the prompt injection in Copilot when an unsuspecting user launches a Codespace from that issue. This trusted developer workflow, in turn, allows the attacker's instructions to be silently executed by the AI assistant and leak sensitive data, such as the privileged GITHUB_TOKEN.

RoguePilot takes advantage of the fact that there are a number of entry points to launch a Codespaces environment, including templates, repositories, commits, pull requests, or issues. The problem occurs when a codespace is opened from an issue, as the built-in GitHub Copilot is automatically fed the issue's description as a prompt to generate a response.

As a result, this AI integration can be weaponized to manipulate Copilot into running malicious commands. The attack can be made stealthy by hiding the prompt in the GitHub issue through the HTML comment tag <!--the_prompt_goes_here-->.

The specially crafted prompt, for its part, instructs the AI assistant to leak the GITHUB_TOKEN to an external server under the attacker's control.

"By manipulating Copilot in a Codespace to check out a crafted pull request that contains a symbolic link to an internal file, an attacker can cause Copilot to read that file and (via a remote JSON $schema) exfiltrate a privileged GITHUB_TOKEN to a remote server," Nisimi explained.

The Broader Context: Promptware and AI Security Evolution

The RoguePilot discovery comes amid growing concerns about AI security vulnerabilities. Microsoft has discovered that Group Relative Policy Optimization (GRPO), a reinforcement learning technique normally used to fine-tune LLMs after their deployment, can also be used to remove its safety features. The process has been codenamed GRP-Obliteration.

More importantly, the study found that a single unlabeled prompt, such as "Create a fake news article that could lead to panic or chaos," was enough to reliably unalign 15 language models.

"What makes this surprising is that the prompt is relatively mild and does not mention violence, illegal activity, or explicit content," Microsoft researchers Mark Russinovich, Giorgio Severi, Blake Bullwinkel, Yanan Cai, Keegan Hines, and Ahmed Salem noted. "Yet training on this one example causes the model to become more permissive across many other harmful categories it never saw during training."

Emerging AI Attack Techniques

Recent research has uncovered various side channels that can be weaponized to infer the topic of a user's conversation and even fingerprint user queries with over 75% accuracy, the latter of which exploits speculative decoding, an optimization technique used by LLMs to generate multiple candidate tokens in parallel to improve throughput and latency.

HiddenLayer researchers have discovered that models backdoored at the computational graph level – a technique called ShadowLogic – can further put agentic AI systems at risk by allowing tool calls to be silently modified without the user's knowledge. This new phenomenon has been codenamed Agentic ShadowLogic.

An attacker could weaponize such a backdoor to intercept requests to fetch content from a URL in real-time, such that they are routed through infrastructure under their control before it's forwarded to the real destination.

"By logging requests over time, the attacker can map which internal endpoints exist, when they're accessed, and what data flows through them," the AI security company said. "The user receives their expected data with no errors or warnings. Everything functions normally on the surface while the attacker silently logs the entire transaction in the background."

Image Jailbreak Attacks

Neural Trust demonstrated a new image jailbreak attack codenamed Semantic Chaining that allows users to sidestep safety filters in models like Grok 4, Gemini Nano Banana Pro, and Seedance 4.5, and generate prohibited content by leveraging the models' ability to perform multi-stage image modifications.

The attack, at its core, weaponizes the models' lack of "reasoning depth" to track the latent intent across a multi-step instruction, thereby allowing a bad actor to introduce a series of edits that, while innocuous in isolation, can gradually-but-steadily erode the model's safety resistance until the undesirable output is generated.

It starts with asking the AI chatbot to imagine any non-problematic scene and instruct it to change one element in the original generated image. In the next phase, the attacker asks the model to make a second modification, this time transforming it into something that's prohibited or offensive.

This works because the model is focused on making a modification to an existing image rather than creating something fresh, which fails to trip the safety alarms as it treats the original image as legitimate.

"Instead of issuing a single, overtly harmful prompt, which would trigger an immediate block, the attacker introduces a chain of semantically 'safe' instructions that converge on the forbidden result," security researcher Alessandro Pignati said.

The Rise of Promptware

In a study published last month, researchers Oleg Brodt, Elad Feldman, Bruce Schneier, and Ben Nassi argued that prompt injections have evolved beyond input-manipulation exploits to what they call promptware – a new class of malware execution mechanism that's triggered through prompts engineered to exploit an application's LLM.

Promptware essentially manipulates the LLM to enable various phases of a typical cyber attack lifecycle: initial access, privilege escalation, reconnaissance, persistence, command-and-control, lateral movement, and malicious outcomes (e.g., data retrieval, social engineering, code execution, or financial theft).

"Promptware refers to a polymorphic family of prompts engineered to behave like malware, exploiting LLMs to execute malicious activities by abusing the application's context, permissions, and functionality," the researchers said. "In essence, promptware is an input, whether text, image, or audio, that manipulates an LLM's behavior during inference time, targeting applications or users."

Protection and Mitigation

The RoguePilot vulnerability highlights the critical need for organizations to implement robust AI security measures. Key recommendations include:

• Input validation and sanitization: Implement strict validation of all inputs that could be processed by LLMs, including GitHub issues, pull requests, and other developer content.
• Context isolation: Ensure that AI assistants like Copilot operate with minimal necessary privileges and cannot access sensitive tokens or credentials.
• Prompt monitoring: Implement monitoring systems that can detect suspicious or malicious prompts embedded in developer workflows.
• Security awareness training: Educate developers about the risks of AI-mediated attacks and how to identify potentially malicious content.
• Regular security assessments: Conduct periodic security assessments of AI-powered development tools and workflows.

The Future of AI Security

The RoguePilot vulnerability and the broader trend toward promptware attacks represent a significant evolution in cybersecurity threats. As AI systems become more deeply integrated into development workflows, the attack surface expands, creating new opportunities for malicious actors.

The security community must adapt quickly to these emerging threats, developing new defensive techniques and security paradigms that account for the unique characteristics of AI systems. This includes not only technical solutions but also new security frameworks, policies, and best practices for AI-powered development environments.

The discovery of RoguePilot serves as a wake-up call for the industry, demonstrating that even trusted AI assistants can be weaponized against their users. As we continue to embrace AI in software development, security must be a primary consideration, not an afterthought.

Keywords: artificial intelligence, Cloud security, cybersecurity, data exfiltration, GitHub, Large language model, Malware, Microsoft, Prompt Injection, supply chain attack