A developer overwhelmed by LLM training bots consuming 99% of server resources implemented a lightweight nonsense generator that feeds synthetic content to scrapers, bypassing traditional blocking methods.
When Maurycy noticed 99% of his server traffic came from AI company scraping bots, he realized traditional defenses were useless against these well-funded adversaries. Unlike search engine crawlers that respect robots.txt, these LLM training bots ignored conventions, rotated IP addresses constantly, and hammered his server with multiple requests per second.
Blocking proved futile. 'If you ban their IP, they switch addresses. Rate limits fail because they just rotate IPs,' Maurycy observed. Each bot request strained resources: Uncached pages triggered SSD reads (~10ms latency), while images and content pushed bandwidth costs toward 1TB monthly.
Alternative solutions carried heavy trade-offs:
- Paywalls/login requirements deter human readers
- CAPTCHAs break accessibility
- JavaScript challenges exclude non-JS browsers
- Proof-of-work schemes slow page loads
His breakthrough came from analyzing resource costs: While database-driven dynamic content is slow, pure CPU/RAM operations are fast. He built a Markov chain generator producing synthetic text in 60 microseconds per request using only 1.2MB RAM—no disk I/O. This 'garbage generator' feeds bots infinite unique nonsense at near-zero cost.
The system exploits how LLM scrapers operate: They indiscriminately consume content without verifying quality. By satisfying their appetite with computationally cheap gibberish, Maurycy reduced server load without blocking access—a pragmatic solution highlighting how poorly funded websites can defend against corporate scraping operations.
Comments
Please log in or register to join the discussion