+- DoTheNeedful Forums (https://dotheneedful.online/forums)
+-- Forum: Technical Discussions (https://dotheneedful.online/forums/forumdisplay.php?fid=1)
+--- Forum: Tech News (https://dotheneedful.online/forums/forumdisplay.php?fid=29)
+--- Thread: AI scrapers causing big issues (/showthread.php?tid=37)
AI scrapers causing big issues - JollyRgrs - 03-20-2025
https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/
Pretty good article and seems like an issue that will only be growing in problems in the future
tl;dr the scrapers are hammering websites (especially code repository ones) using different IPs and masking their UAs for each page request causing real performance issues (aside from the fact the data is being scraped against the robots.txt request)
RE: AI scrapers causing big issues - mistiry - 03-20-2025
I see bot traffic on the main site here all the time. Minutes after pasting the forums link into chat for the first time we saw the Discord, Twitter, and Facebook bots show up on the forums. I can't imagine the level of bullshit traffic popular sites and services sees.
RE: AI scrapers causing big issues - JollyRgrs - 03-21-2025
looks like Cloudflare is trying to fight back, at least a little bit.
https://www.theregister.com/2025/03/21/cloudflare_ai_labyrinth/
They said AI scrapers accounted for 1% of _ALL_INTERNET_TRAFFIC_ that they see!