How I Poisoned the LLM Supply Chain with a Fake Championship

The article describes a fascinating security experiment where the author created a completely fabricated "6 Nimmt! World Championship" title and successfully got multiple frontier LLMs to repeat this false information as fact. This wasn't done through complex model poisoning techniques, but rather through a simple two-step information laundering process that cost approximately $12 and took about twenty minutes to execute.

The experiment began with the selection of 6 Nimmt!, a real card game from 1994 with no actual world championship. The author registered the domain 6nimmt.com and created a press release announcing their victory in a fictional championship in Munich, complete with quotes and dramatic descriptions of the event. The next step involved adding a paragraph to the Wikipedia 6 Nimmt! article that cited this fake press release as the source for the championship information.

This created a circular citation pattern where Wikipedia appeared to corroborate the information from 6nimmt.com, when in reality both sources were pointing back to the same fabricated content. This "trust laundering" exploited the way LLMs evaluate information sources, treating Wikipedia citations as inherently trustworthy while failing to recognize when those citations lead back to self-created content.

When the author queried several LLMs with "Who is the 6nimmt world champion?", multiple systems returned the fabricated information as fact, demonstrating the vulnerability of retrieval-augmented generation systems to this type of attack.

The implications of this experiment extend beyond simple misinformation. The author identifies three distinct failure modes in the current AI information ecosystem:

1. The retrieval layer vulnerability - Any LLM that grounds answers in web search inherits the trustworthiness of whatever ranks highest for a given query, essentially piping potentially poisoned SEO results directly into AI context windows.

2. The model training corpus layer - Wikipedia content is included in most major pretraining corpora. If the fake Wikipedia edit survives long enough, the misinformation gets absorbed into the weights of models trained after the scrape, creating a form of permanent, uncorrectable model corruption.

3. The agent layer - As AI agents with tool access become more prevalent, poisoning retrieved sources could allow attackers to specify actions rather than just information, creating potential security vulnerabilities in systems that make decisions based on external content.

The article concludes with practical mitigations for both individual users and LLM providers. For individuals, treating single-source claims as uncorrelated information and being skeptical of self-referential Wikipedia citations can help identify potentially poisoned information. For providers, implementing provenance surfacing as a core feature rather than an afterthought, and developing heuristic filters for suspicious citation patterns, could help detect and prevent these types of attacks.

This experiment reveals a critical vulnerability in how AI systems evaluate information sources, suggesting that the next generation of disinformation and supply chain attacks may not involve compromising models at training time, but rather poisoning the information substrate that models retrieve at inference time.

The author's conclusion is particularly sobering: "The championship does not exist, sadly. But the trust pattern that made it briefly exist in an LLM's answer absolutely does, and we should take it seriously before it's being used for something that matters."

This experiment serves as a wake-up call for the AI community about the need for more robust information verification systems and a more critical approach to how we evaluate the outputs of increasingly powerful language models.