AI scrapers causing big issues

[JollyRgrs]

https://thelibre.news/foss-infrastructur...companies/

Pretty good article and seems like an issue that will only be growing in problems in the future

tl;dr the scrapers are hammering websites (especially code repository ones) using different IPs and masking their UAs for each page request causing real performance issues (aside from the fact the data is being scraped against the robots.txt request)

[mistiry]

I see bot traffic on the main site here all the time. Minutes after pasting the forums link into chat for the first time we saw the Discord, Twitter, and Facebook bots show up on the forums. I can't imagine the level of bullshit traffic popular sites and services sees.

[JollyRgrs]

looks like Cloudflare is trying to fight back, at least a little bit.

https://www.theregister.com/2025/03/21/c...labyrinth/

They said AI scrapers accounted for 1% of _ALL_INTERNET_TRAFFIC_ that they see!