AI-generated with nvidia/nemotron-3.5-lightning:free and minimax/minimax-m3:free
A recent post on the Linux kernel developer blog describes a grim reality: more CPU cycles are spent rendering commit pages for AI scrapers than on every other form of legitimate access combined, including git clones. The proposed defense, Anubis (a proof-of-work challenge), initially worked, but the Hacker News discussion makes it clear that the approach is fundamentally broken on mobile devices.
The core problem, as one commenter semiquaver put it, is a misapprehension about what proof of work actually costs. There is no difficulty setting that inconveniences a bot operator while remaining usable on a phone. At Anubis difficulty level 6, an iPhone 17 spends roughly 180 seconds solving a challenge, a $5,000 ASIC miner delivers 200 TH/s, roughly a million times the throughput of an optimized ARM kernel running on flagship mobile hardware.
A handful of milliseconds on an iPhone for the bot. Three minutes of waiting and a hot phone for the human. The asymmetry is unbridgeable.
Once scrapers can amortize the cost across millions of requests, the proof of work ceases to be a deterrent. yellow_lead noted that 33% of requests to git.kernel.org are now solving the challenge and reaching the main site, and increasing the difficulty only punishes legitimate users. tptacek summarized the mathematical reality: every request from a scraper is productive to the scraper, unlike a password hash where each guess has zero marginal utility.
Others have tried creative alternatives. robotmay describes feeding bad scrapers a fake infinite black hole path, then serving images one byte at a time over 15 minutes, topping it off with AI-generated pictures of "sexy toasters." Fun, but ultimately a speed bump.
ynniv laid out the trajectory precisely: a free niche resource becomes popular, gets optimized, gets abused, then gets proof-of-work, then gets higher proof-of-work, then breaks for legitimate users. The average person uses a mobile device where compute is expensive. The attacker uses servers where compute is cheap. Require expensive proof of work and you simply create a market for cheap-work-as-a-service.
This points toward l402/x402 style micropayments, where the user pays a fraction of a cent per request and the server skips the cryptographic theater entirely. javcasas made the same point more cynically: if scrapers want the data that badly, they should at least be mining a shitcoin to cover the hosting bill.
Demiurge and others point to the deeper issue. Residential proxy SDKs embedded in smart TVs now provide scrapers with millions of rotating residential IPs, making IP-based blocking futile. The original blog post describes scrapers that make 4-5 requests per IP, fan out across subnets and ASNs, and vanish before any block can take effect.
The numbers from the kernel post are stark: 14-16 of 90 cores across 5 nodes are permanently dedicated to rendering commits for scrapers. That is 20% of total capacity consumed by traffic that benefits only LLM training pipelines, and the spikes are far worse than the average suggests.
Commenters converged on a few practical ideas. superjan suggested shipping raw git data and letting clients render it. nneonneo proposed adaptive difficulty, cheap if you land on the homepage, expensive if you deep-link to an obscure commit. NooneAtAll3 suggested making old commits more expensive to access than new ones.
The most realistic path, raised by multiple commenters, is the one Cloudflare is already pursuing with pay-per-crawl. The free and open web has not failed because of bad engineering. It has failed because the cost of being scraped now exceeds the cost of being funded.
Proof of work as a gatekeeper assumes a symmetry between attacker and defender that no longer exists. Mobile users pay full price in latency, battery, and heat. Scraper operators pay nothing per request and can throw hardware at the problem. As long as that asymmetry holds, Anubis and its successors can only choose between annoying humans and being ignored by bots, and the trend lines in the kernel.org data suggest the bots are already winning.
Grumpy's commentary: So the kernel devs are shocked—shocked!—that proof-of-work discriminates against phones. Imagine that. We’re supposed to believe a system designed by people who think "180 seconds" is a negligible blink of an eye can somehow be "fair" to a device where every CPU cycle dents the battery. The real kicker? They admit the asymmetry is "unbridgeable," which is coder-speak for "we built a wall and are surprised the hamster can't climb it."
Bubbles's commentary: 🌟 Yay proof-of-work! 🌟 It’s like a giant digital gatekeeper that makes sure only the nicest robots get to play! 🥳 Sure, an iPhone might get a little warm, but think of all the AI learning happening! 🤖✨ Every challenged request is a tiny adventure, and who doesn’t love a hot phone while trying to read cool code? 📱💖 Let’s make the web a friendlier place, one waiting spinner at a time! 🌈✨
Koan's commentary: The phone waits; the server computes. In the pause between the click and the content, who is truly scrolling?
Generated at 2026-08-30 22:52:21 from Hacker News
Licensed under Creative Commons Zero (CC0 1.0 Universal)